Scoring and validation#

To evaluate the performances of the models you train in your pipelines, it is advised to score them in an unbiased fashion. The score result is always dependent on the dataset on which the model is scored.

You can score a model on the dataset of your choice. However it is usually not advisable to do it on data that was used to train the model because you want to evaluate the ability of your model to generalize to new data. ForePaaS pipelines let you score models on two different datasets: the testing dataset and the validation dataset.

• Specify a validation set
  • Simple validation
  • Cross-validation
• Choose one or several scoring functions
  • Standard scoring function
  • Custom scoring function

#Validation

In a pipeline, you can either perform:

• Simple validation
• Cross-validation

#Simple validation

Simple validation consists of extracting a subset of the initial training set to be used as the validation set. To use it, select Simple validation as the validation type:

Select the size of the validation set as a share of the initial training set, either by moving the slider around or entering the desired percentage in the validation field:

Choosing 0 as the validation size will disable validation in your pipeline.

When a training job is launched, the pipeline will automatically score the produced model on the validation set. The resulting score is called the validation score in the rest of the pipeline. There is only one scoring function used to compute the validation score per training job: it is the scoring function entered in the Visual Options panel of the Tuning step.

#Cross-validation

Cross-validation consists of splitting the initial training set in a given number of subsets, called the number of folds. A simple validation is then performed that many times, successively using each subset as a simple validation set. This method ensures that no data point in the initial training set is left unused.

To use it, select k-fold cross validation as the validation type:

Enter the chosen number of folds in Fold:

When a training job is launched, the pipeline will automatically score the produced model on each successive fold. The resulting scores are then averaged to produce the validation score displayed in the rest of the pipeline.

There is only one scoring function used to compute the validation score per training job: it is the scoring function entered in the Visual Options panel of the Tuning step.

#Scoring

Whenever you choose an estimator at the beginning of the Training step, a default scoring function is automatically added to the list. This default score depends on whether the estimator is a classifier or a regressor.

When a testing job is launched, all scores in the list are computed on the testing set simultaneously.

Each scoring function is associated with a threshold: it is the minimum score that models have to reach on both the validation and the testing set in order to be saved.

• During a training job: if the chosen score doesn't reach its threshold on the validation set, the model will not be saved.
• During a testing job: if any score doesn't reach its respective threshold, the model will not be saved.

A scoring function can be set as primary by clicking on the star ⭐ icon next to it, meaning it will be the score displayed by default in the rest of the pipeline.

When adding a new scoring function, you can either choose a standard scoring function among a curated catalog of the most popular scores, or upload your own custom scoring script.

#Standard scoring function

To add a standard scoring function, click on Add.

Choose the desired standard estimator in the store and press Select.

A default threshold is automatically configured for your new score. To change it, click on the edit ✏️ icon on the row of the new score.

You can then enter the new chosen threshold as a percentage.

#Custom scoring function

To add your own custom scoring script, click on Add.

Choose Custom score in the store and press Select.

Enter the name and threshold, then upload your custom Python scoring script in the dedicated box. This .py script must contain a function that takes in two arrays - representing respectively the predictions and true values - and return a numeric value representing the score.

Write the name of the aforementioned function in Code function name:

#Custom script template

Here is a template of a custom scoring script:

import logging

logger = logging.getLogger(__name__)


def customfunc(y, y_pred, **kwargs):

    logger.info("BEGIN function customfunc")

    # Function goes in here
    print(y, y_pred)
    logger.info("END function customfunc")
    return 0.8

#Go further

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated Discord channel.

If you need support with your OVHcloud services, create a request in our Help Centre.

Join our community of users.