Lesson 5 · about 12 minutes

Prepare data so the model learns the right problem.

Dataset preparation defines what counts as an example, what information the model may use, and whether your evaluation can be trusted.

Dataset, example, feature, label

A dataset is a collection of examples. One example might be a house sale, an image, or a text document. Features are the inputs used to make a prediction. A label is the desired answer. For a spam classifier, email text and sender domain can be features; “spam” is the label.

Features must exist at the moment you will actually make a prediction. “Whether a customer eventually cancelled” cannot be a feature for predicting cancellation; that would leak the answer.

Split before you learn from the data

Use a training split to fit model weights and preprocessing choices. Use validation to choose among models, settings, and stopping points. Keep the test split untouched until the final evaluation. If preprocessing learns a statistic—such as an average used for normalization—fit it only on training data, then apply it to validation and test.

Random splits are not always valid. For repeated users, products, documents, or near-duplicates, keep each group in one split. For data that changes over time, train on earlier data and evaluate on later data. Otherwise, the model can get an unrealistically easy preview of the future.

Cleaning is a policy, not just deleting rows

Write down why every filtering rule exists. Typical checks include: permitted sources and licenses; personal or secret data; corrupt or empty records; duplicate and near-duplicate content; language/domain balance; annotation quality; and known harmful or irrelevant material. Deduplication often needs to compare the full corpus before splitting so duplicates cannot land in separate splits. Aggressive cleaning can improve quality but also remove minority dialects, rare cases, or needed domain examples—so inspect what was removed.

What “features” means for LLMs

For classical ML, you may engineer features: word count, account age, price, or a category encoded as numbers. For an LLM, raw text is tokenized. During pretraining, the input tokens are usually [t₁, t₂, …, tₙ₋₁] and labels are the same sequence shifted left, [t₂, t₃, …, tₙ]. The model learns its own rich internal features (representations) from those tokens. An attention mask records which positions are real content rather than padding; it is control information, not a linguistic feature.

A reproducible preparation pipeline

1. Define task and one valid example.
2. Record source, license, collection date, and intended use.
3. Apply documented global deduplication, then split using a documented seed, group rule, or time boundary.
4. Apply versioned cleaning rules.
5. Fit any learned transform on training only, then transform validation and test.
6. Transform inputs: feature encoding for tabular ML; tokenization and label shifting for LLMs.
7. Audit examples and distributions in every split; document limitations.

Use the printable dataset-preparation checklist. Primary source: “Datasheets for Datasets” by Gebru et al.