Gradient Descent and Optimizers

Objective: tune optimization without confusing learning speed with quality.

Gradient descent updates the parameters in the direction opposite to the gradient:

θ ← θ - η ∇L(θ)

η is the learning rate. Too small: slow progress or stagnation. Too large: oscillation, divergence or NaN.

The main optimizers

• SGD: direct update; simple and often excellent generalization.
• Momentum: averages recent directions and reduces oscillations.
• Adam: adapts the step per parameter; efficient and fast baseline.
• AdamW: correctly decouples weight decay; widely used with Transformers.

optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=1e-2)