Regularization and Generalization

2 min

Objective: reduce the gap between train memorization and real-world performance.

Generalization is measured on examples not used to fit the parameters nor to choose the hyperparameters. An excellent train and a degraded validation signal overfitting, but distribution mismatch or a pipeline bug must also be ruled out.

Main levers

  • More useful data: often the most robust lever.
  • Augmentation: transformations that preserve the label.
  • Weight decay: penalizes excessive weights.
  • Dropout: randomly masks activations during training.
  • Early stopping: keeps the best validation state.
  • Reducing capacity: fewer layers, channels or units.
python
dropout = torch.nn.Dropout(p=0.3)

Preview — the rest of the lesson is for enrolled readers.

Already enrolled with a code?

Your access is tied to your account, not to this link. Sign in with the same email you used in class: your course is waiting, no need to enter the code again.

Sign inNo account yet? Create one
This lesson is part of the “Training the Network” module

The first modules of the course are open to everyone. For the rest you have three options: buy this course once and for all, subscribe, or enter the code handed out in class.

Subscribe — 19 $ US/monthBack to the outline
Are you a student on this course?

The code is tied to your account: sign in or create an account and it will be applied automatically when you come back.

No account yet? Create one