Objective: understand capacity, composition and representation learning.
A deep network composes functions:
f(x) = f_L(f_{L-1}(...f_2(f_1(x))))Each layer transforms the received representation. With non-linear activations,
this composition can approximate complex relationships. Without non-linearity,
several dense layers reduce to a single affine transformation: W3(W2(W1x)) = Wx.
A feature is no longer necessarily a hand-crafted column. In a CNN, a hidden channel may react to a diagonal; further along, a combination of channels may react to a wheel. The meaning is not programmed explicitly: it emerges if that organization reduces the loss function.
One specialist checks edges, another groups textures, a third recognizes parts. None decides alone; their composition produces the final decision. In reality, the "specialists" are matrices of numbers, adjusted together.
Increasing capacity sometimes reduces underfitting, but raises cost and the risk of memorization. The right question is: "what capacity generalizes under my constraints?"
On modest tabular data, boosted trees often remain highly competitive. If the signal is weak, adding layers creates no information. If the labels are wrong, the network can learn those errors with impressive efficiency.
Why do you need a non-linear activation between two layers? What is the difference between width and depth? What happens to a network if the inputs contain no signal?
Without non-linearity, the composition stays affine. Width counts the units of a stage, depth counts the stages. Without signal, the network can only learn randomness or artifacts.
Sketch two models with comparable parameter budgets: a single wide layer and three narrower layers. Describe a composed function that the second represents naturally and specify the control that would distinguish a depth gain from a mere compute gain.
Reading: each stage transforms the received representation and makes certain factors easier to separate. Depth helps when this composition matches the structure of the problem. The main trap is confusing "more layers" with "more quality": without data, regularization and an honest protocol, extra capacity mostly increases the risk of overfitting.