- Block 9 — Generalization
- Objective
- define overfitting and underfitting by the relation between
two risks — the empirical risk on the fitting sample and the true risk under
the distribution — rather than by the informal slogan "the model learned the
noise"; connect both conditions to the capacity of the hypothesis space and to
its effective, not nominal, size; produce the classic U-shaped test-error
curve from real measurements on three different capacity knobs; build learning
curves and read high bias against high variance from their shape; separate the
validation curve from the learning curve, which answer different questions;
demonstrate memorization by fitting random labels; place the interpolation
threshold and double descent as a measured caveat to the classic U; expose the
slower failure in which the validation set itself is overfitted by repeated
selection; recognize underfitting hidden behind a metric that has saturated;
and read the signature of each condition off a table of train, validation and
test numbers.
- Estimated duration
- 70 minutes
- Prerequisites
- chapters 026 (train/validation/test split), 028 (data
leakage), 029 (fit, predict, predict_proba) and 030 (error, loss and metric)
- Associated quizzes
031.1-quiz-generalization.md to
031.7-quiz-capacity-and-regularization.md