Quick reference

Dataset preparation checklist

1. State the task: prediction target, intended users, and unacceptable errors.
2. Inventory sources: license, consent, provenance, date range, language/domain mix, and sensitive-data risks.
3. Define one example: inputs (features), target/label if any, identifier, and metadata.
4. Split early: train for fitting, validation for decisions, test for one final unbiased estimate. Group or time-based splits may be required.
5. Clean reproducibly: remove corrupt items and PII as appropriate; normalize only using training-set statistics; deduplicate before or within split policy.
6. Transform: encode categories/numbers for conventional ML; tokenize and create shifted labels/masks for LLMs.
7. Audit: inspect examples, distributions, duplicates, leakage, coverage, and benchmark contamination.
8. Version and document: preserve code, dataset version, split seed/rules, and a datasheet.

Feature vs. learned representation

In conventional ML, a feature is usually an input column deliberately selected or engineered, such as price, age, or word count. In an LLM, tokens are inputs; the useful internal features are learned representations produced by the model's layers, not a hand-built spreadsheet of language properties.

Sources: Gebru et al. (2021); Hugging Face Datasets documentation.