- Block 7 — Train, validation and test
- Objective
- establish why an error measured on the data that built a model
carries no information about generalization; separate the roles of the three
sets; distinguish model selection from model assessment and show why
one set cannot perform both; master
train_test_split and the semantics of
random_state; size the partitions from the precision required on the
announced figure; quantify the variance of a single split and reduce it with
cross-validation; know the splitter family and the bias–variance trade-off of
k; run a nested cross-validation; reserve hold-out data for threshold tuning
and calibration; and recognize a contaminated test set. - Estimated duration
- 70 minutes
- Prerequisites
- chapters 005 (dataset structure), 008 (algorithm and
model), 009 (parameters and hyperparameters), 012 (life cycle, stage 8) and
025 (feature selection)
- Associated quizzes
026.1-quiz-biased-training-error.md to
026.8-quiz-nested-cv-and-contamination.md