Lesson 4 · about 14 minutes

Two layers: how each one gets its own correction

The output error travels backward through the network. Each weight receives the portion of that error it was locally responsible for.

The tiny network

input x ──× w₁──→ hidden value h ──× w₂──→ output o target y ──────────────────────────────────────→ loss L = (o − y)²

There is one hidden layer containing one number. We deliberately use no activation function, bias, or multiple neurons yet, so you can see the mechanism. The forward equations are h = w₁x, then o = w₂h. The loss says how far the output is from the target.

Forward pass: make the error visible

Use x = 2, w₁ = 1, w₂ = 3, target y = 12. First layer: h = 1 × 2 = 2. Output layer: o = 3 × 2 = 6. Loss: (6 − 12)² = 36. Both weights helped create the too-small answer, but in different places.

Backward pass: start at the output

The loss slope with respect to output is dL/do = 2(o − y) = −12. Negative means that increasing the output would reduce loss. For the output-layer weight, do/dw₂ = h = 2, so dL/dw₂ = −12 × 2 = −24. Gradient descent therefore increases w₂.

Pass the signal into the hidden layer

The hidden value affects loss only by affecting the output. Its error signal is dL/dh = (dL/do)(do/dh) = −12 × w₂ = −36. Now link hidden value to the first weight: dh/dw₁ = x = 2. Therefore dL/dw₁ = −36 × 2 = −72. The first-layer weight gets a larger correction because changing it influences the hidden value, which is then multiplied by w₂.

This is the essential insight: a hidden layer has no target label of its own. Backpropagation gives it credit or blame by carrying the output error backward through all downstream operations.

Update both weights together

With learning rate η = 0.01, update using the gradients computed from the old weights: w₁ = 1 − 0.01(−72) = 1.72; w₂ = 3 − 0.01(−24) = 3.24. The new output is 3.24 × (1.72 × 2) = 11.1456. New loss is about 0.73, sharply down from 36. Real optimizers take more cautious, batched steps—but the logic is identical.

Explore the calculation

Scaling to real networks

With many neurons, each layer uses matrices rather than single weights. A gradient is then a matrix of the same shape as the weight matrix. Each neuron sums contributions from the next layer; matrix multiplication performs those sums efficiently. Activations, attention, and normalization add operations, but every one supplies a local derivative. Automatic differentiation chains them backward.

Retrieval check

In this network, which gradient directly contains the downstream weight w₂: dL/dw₁ or dL/dw₂? Enter w1 or w2.

Keep the two-layer reference with the earlier backpropagation reference. Primary source: Rumelhart, Hinton & Williams (1986).