Multilayer Perceptron and Hidden Representations

Objective: build a mental model of an MLP and count its parameters.

An MLP is a feed-forward network made of fully connected layers.

a⁽⁰⁾ = x
z⁽ˡ⁾ = W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾
a⁽ˡ⁾ = φ(z⁽ˡ⁾)

A Linear(p, q) layer contains p × q weights and q biases, that is (p + 1)q parameters. A model 784 → 128 → 10 therefore contains 784×128+128 + 128×10+10 = 101,770 parameters.

from torch import nn

model = nn.Sequential(
    nn.Flatten(),
    nn.Linear(28 * 28, 128),
    nn.ReLU(),
    nn.Linear(128, 10),
)

Hidden layers

"Hidden" means its output is neither the raw input nor the delivered prediction. It builds an intermediate representation. Do not assume that each neuron has a simple human meaning; a representation is often distributed across several units.

Output layer

The output depends on the task:

A logit is an unnormalized score. In multiclass classification, PyTorch generally expects the logits directly; CrossEntropyLoss applies the appropriate numerically stable transformation.

Pitfalls

• Adding softmax in the model before CrossEntropyLoss.
• Counting the input layer as a parameterized layer.
• Picking hundreds of millions of parameters without data or justification.

Quick check

How many parameters does Linear(20, 5) contain? What output dimension for 7 exclusive classes?

Mastery activity — Parameter budget

Compute the shapes and parameters of an MLP 20 → 8 → 5 → 3, including biases. Then propose a variant that is twice as small and specify what you keep constant to compare the two. Success: each matrix is oriented correctly and the protocol never touches the test set.

Propagation of dimensions in an MLP

Reading: a linear layer changes the last dimension, never the batch size. For a K-class classification, the last layer produces exactly K logits. The main trap is applying an output activation incompatible with the loss function, for example softmax before CrossEntropyLoss.