The last section demonstrated creation of training and testing dataset for the ML process.
Its not adviced to use the test set to assess model performance during the training phase.
How do we assess the generalization performance of the model?
- One option is to assess an error metric based on the training data. Unfortunately, this leads to biased results as some models can perform very well on the training data but not generalize well to a new data set.
Use a validation approach, which involves splitting the training set further to create two parts: a training set and a validation set (or holdout set).
We can then train our model(s) on the new training set and estimate the performance on the validation set.
Validation using single holdout set can be highly variable and produce unreliable results.
- Solution: Resampling Methods
Resampling Methods: Alternative approach, allowing for repeated model fits to parts of the training data and test it on other parts. Two most commonly used methods are:
K-fold Cross validation
Let’s briefly discuss these two.