Lesson 2 · about 9 minutes

How an LLM changes its weights

An LLM improves through millions of tiny corrections—not by being handed a list of rules or facts.

1. Start with a prediction task

Training begins with text collected from many sources: web pages, books, code, reference material, and licensed or human-created datasets. A serious pipeline considers permission, provenance, privacy, quality, language balance, duplication, and harmful content. The exact mixture is a major design choice; for many open-weight releases, it is only partly disclosed.

After filtering and deduplication, a tokenizer turns each document into token IDs. The stream is divided into fixed-length sequences. For [The, cat, sat], the model is trained to predict cat after The, then sat after The cat. One sequence supplies many prediction targets.

2. Measure error: loss

At first, weights are usually randomly initialized, so the correct next token may receive little probability. The loss function penalizes this. For a correct token with probability p, a common contribution is −log(p): confidently wrong predictions are punished much more than uncertain ones. The training objective is the average loss over every target token in a batch.

3. Turn error into a correction

Backpropagation applies calculus to determine how a tiny change to each weight would affect loss. This produces gradients. An optimizer—commonly AdamW for transformer LLMs—combines those gradients with running statistics and makes a small update. A new batch arrives, and the process repeats on enormous clusters, often for trillions of tokens.

The model does not store each sentence as one neat record. Its behavior changes because distributed numerical patterns across many weights make useful continuations more likely.

4. How teams decide it improved

Training loss normally falls, but teams also evaluate on held-out data and task benchmarks. They check memorization, degraded performance in a language or domain, safety failures, and instability. Lower loss means the next-token task improved on average; it does not guarantee that every answer is correct.

5. Why pretraining is not the end

Pretraining gives broad language competence. To make a model follow requests, developers commonly post-train it: supervised fine-tuning on demonstrations, then preference-based training from human or model feedback. This shapes helpfulness and safety, and can trade off against raw breadth of capability.

Use the training-pipeline reference and glossary. Primary sources: Vaswani et al. (2017) and Ouyang et al. (2022).