Objective: align target, output layer and loss function.
The loss provides the training signal. The metric describes quality according to the need. A loss must be differentiable almost everywhere; a business metric need not be.
| Task | Output | Target | PyTorch loss |
|---|---|---|---|
| Regression | (N,1) or (N,) | float | MSELoss, L1Loss, HuberLoss |
| Binary | 1 logit | 0/1 float | BCEWithLogitsLoss |
| K classes | K logits | integer index | CrossEntropyLoss |
| Multilabel | K logits | K values 0/1 | BCEWithLogitsLoss |
For the true class y, the multiclass loss equals -log p_y. A confident
and wrong prediction is heavily penalized. CrossEntropyLoss combines
log_softmax and the negative log-likelihood; do not feed it probabilities
already normalized.
The loss of a batch is generally an average. With rare classes, you can weight the classes, but weighting replaces neither an appropriate validation protocol nor metrics such as recall, precision and PR-AUC.
import torch
from torch import nn
weights = torch.tensor([1.0, 4.0])
criterion = nn.CrossEntropyLoss(weight=weights)Label smoothing replaces a fully certain target with a slightly softened distribution. It can reduce over-confidence, but changes calibration and must be validated, not applied ritually.
Why is accuracy not used directly as a loss? Should softmax be applied before
CrossEntropyLoss?
Accuracy is discontinuous and provides almost no gradient. No, the loss expects logits.
For real-estate pricing, binary fraud, one species among ten and independent tags, specify the logit shape, target encoding, training loss and business metric. Add a case where MAE would be preferred over MSE. Success: the loss and the metric play explicitly distinct roles.
Reading: the loss makes learning differentiable, while the metric judges the value of the model. They can therefore be different. The main trap is choosing a loss just because its name resembles the metric, without checking the target format, expected logits and business cost of errors.