Lesson 3 · about 12 minutes

Backpropagation tells every weight how to change.

It is not a separate kind of intelligence. It is an efficient way to calculate the effect of each weight on one observed error.

The goal: lower loss

Take a deliberately tiny model: one input x, one weight w, and prediction ŷ = w × x. If the target is y, define loss L = (ŷ − y)². Loss is a score: zero is perfect for this example; larger values are worse.

Suppose x = 2, w = 1, target y = 6. The forward pass gives ŷ = 2, then L = (2 − 6)² = 16.

The key question

“If I nudge w upward a little, will the loss rise or fall, and how quickly?” The answer is the derivative ∂L/∂w, called the gradient for that weight. A positive gradient means increasing that weight raises loss locally, so gradient descent moves it down. A negative gradient means increasing it lowers loss locally, so the update moves it up.

Why the backward pass works

The weight affects loss through a chain: w → ŷ → error → L. The chain rule multiplies each local influence:

In the example, 2(2 − 6) × 2 = −16. With learning rate 0.1, the update is w ← 1 − 0.1 × (−16) = 2.6. Its next prediction is 5.2, closer to 6, so loss falls. This is one optimizer step.

Try the numbers

What changes in a real LLM

An LLM has billions of weights and many operations: matrix multiplications, attention, normalization, and nonlinear functions. The same rule applies. Autodifferentiation software records forward computations, then backpropagates derivatives in reverse order. It gets gradients for all weights in roughly the same order of computation as the forward pass—not one full re-run per weight. Gradients are averaged across a batch; AdamW then uses them to update weights.

Important limits

Backpropagation does not choose training data, invent the objective, or prove a model is truthful. It only answers: given this loss and this batch, which small local weight changes reduce the loss? The objective, data quality, learning-rate schedule, regularization, and evaluations determine whether those local corrections produce a useful model.

Save the backpropagation reference. Primary source: Rumelhart, Hinton & Williams (1986). LLM context: Vaswani et al. (2017).