Objective: keep activations and gradients within a usable range.
All weights must not be initialized to zero: the neurons of the same layer would stay symmetric and learn the same thing. Random initialization breaks this symmetry, but its scale must preserve the signal.
tanh.PyTorch already initializes the usual layers reasonably. A manual initialization must have a justification and be recorded.
BatchNorm normalizes activations using mini-batch statistics, then learns a
scale and a shift. In validation, it uses learned running averages. That is
why forgetting model.eval() changes the results.
Preview — the rest of the lesson is for enrolled readers.
Your access is tied to your account, not to this link. Sign in with the same email you used in class: your course is waiting, no need to enter the code again.
Sign inNo account yet? Create oneThe first modules of the course are open to everyone. For the rest you have three options: buy this course once and for all, subscribe, or enter the code handed out in class.