Objective: choose an activation and understand saturation, gradients and output.
| Activation | Expression/idea | Use and caution |
|---|---|---|
| ReLU | max(0, z) | simple, fast; dead neurons possible |
| LeakyReLU | small slope if z<0 | reduces permanently zero units |
| tanh | output [-1, 1] | centered, but saturates at extremes |
| sigmoid | output [0, 1] | mainly binary output; saturation |
| GELU | smooth modulation | common in Transformers |
The derivative of ReLU is 1 for z>0 and 0 for z<0. A unit that always
receives negative pre-activations no longer transmits any gradient: it is
"dead".
BCEWithLogitsLoss.CrossEntropyLoss.softmax(z_i) = exp(z_i) / Σ_j exp(z_j) turns K logits into positive numbers
that sum to 1. Adding the same constant to all logits does not change the
probabilities. Stable implementations subtract the maximum before the
exponential.
Use ReLU as a baseline for an MLP/CNN, GELU for a Transformer, and only place an output activation if the loss or interface explicitly requires it.
Why must activations be non-linear? Which activation produces exclusive probabilities?
Otherwise the entire network reduces to an affine transformation. Softmax produces a distribution over exclusive classes.
Compute ReLU and sigmoid for z = {-2, 0, 2}, then the stable softmax of
[1000, 1001, 999]. Then match an output to four tasks: binary, multiclass,
multilabel and unbounded regression. Success: no double activation before a
logit-based loss.
Reading: the loss function often dictates the expected numerical format. With stable losses that take logits, sigmoid or softmax are for interpretation, not for the loss input. The main trap is therefore the double activation, which degrades gradients and can give misleading probabilities.