Quick reference

Backpropagation: the compact algorithm

1. Forward pass: compute every intermediate value from inputs and current weights.
2. Loss: compare output with the target using a differentiable objective.
3. Backward pass: start with how loss changes with the output, then apply the chain rule backward through each operation.
4. Gradient: obtain one derivative for every trainable weight, ∂L/∂w.
5. Update: an optimizer changes weights approximately as w ← w − learning-rate × ∂L/∂w.
6. Repeat: average over a batch and run many steps; reserve held-out data to evaluate generalization.

Three guardrails

The gradient is a local direction, not a guarantee of global perfection. The learning rate controls step size: too large can diverge; too small learns slowly. Backpropagation computes gradients; the optimizer decides the update rule.

Primary source: Rumelhart, Hinton & Williams (1986).