Forward Propagation: From Input to Prediction

Objective: follow the computation of a network and distinguish training from inference.

Forward propagation (forward pass) applies each layer in order. It produces the logits, then the loss if the targets are available. It does not modify any weights.

logits = model(x)          # forward pass
loss = criterion(logits, y)

For a batch x of shape (32, 20), a layer Linear(20, 64) gives (32, 64). A second Linear(64, 4) gives (32, 4). The batch axis is preserved; the last axis changes with the layer.

Training and inference

In training, the network keeps the information needed to compute gradients. In inference, that tracking is disabled and the model is placed in evaluation mode:

model.eval()
with torch.inference_mode():
    logits = model(x)
    predictions = logits.argmax(dim=1)

model.eval() sets the behavior of Dropout and BatchNorm. inference_mode() avoids building the gradient graph. The two operations are complementary.

A traceable computation

A network is a graph of operations. PyTorch records which operations produced each tensor when requires_grad=True. This trace will be traversed in reverse by backpropagation.

Pitfalls

• Using argmax before the loss function: the operation destroys the information useful to the gradient.
• Forgetting model.eval() during validation.
• Computing validation with gradient tracking and wasting memory.

Quick check

Does forward propagation learn? Why call both eval() and inference_mode()?

Mastery activity — Inference contract

Write a pseudo-test that calls the same model containing dropout twice, first in training mode then in evaluation mode. Predict which results must be identical and explain the distinct role of eval() and inference_mode(). Deliverable: test and diagnosis.

Training and inference: two distinct paths

Reading: the forward pass computes the outputs in both cases, but only the training path keeps what is needed to compute gradients. In inference, combine model.eval() and torch.inference_mode(). The trap is thinking one replaces the other: they control different mechanisms.