Once you have decided to finetune, the model and the training loop are the easy part: frameworks handle them. What decides whether the project succeeds is the dataset. Dataset engineering is the work of deciding what data you need, getting it (from users, annotators, public sources or a model), checking it, and turning it into the exact shape the training pipeline expects. It is unglamorous and it is where most of the engineering time goes.

This chapter follows the book's three movements. Data curation asks what data to collect and how to judge it: quality, coverage, quantity, and how to acquire and annotate it. Data synthesis covers the techniques for generating data programmatically or with a model, how to verify what you generate, and the known failure modes of training on model outputs. Data processing is the pipeline that turns a pile of examples into a training set: inspect, deduplicate, clean and filter, format. Throughout, the standard to hold the data to is simple: it should look like what the model will see in production, and it should show the behaviour you want.

In one minute

• Training data should mirror production: same prompt shape (system prompt, retrieved context, tool results), same distribution of tasks, same output format. Mismatch here is the most common silent failure.
• Judge a dataset on three axes. Quality: each example is relevant, consistent, correctly formatted, unique and compliant. Coverage: the examples span the tasks, topics, lengths and styles the model will meet. Quantity: enough to learn, which depends on the method (PEFT needs less), the task and the base model.
• Run the small-data experiment before buying more data: finetune on 25%, 50% and 100% of what you have and look at the curve. Flat means more of the same will not help.
• Synthesis is worth it for quantity, coverage, quality, privacy and distillation. Rule-based and simulation methods are cheap and controllable; AI-powered methods (Self-Instruct, Evol-Instruct, back-translation, judge-ranked preference data) scale but need verification.
• Verify synthetic data with the strongest check available: execution for code and maths, an AI judge for open text, heuristics for bulk filtering, and a human sample to calibrate all three.
• Four known hazards of AI-generated data: quality control at scale, superficial imitation of a teacher's style without its competence, model collapse under recursive training, and lost data lineage.
• Processing order matters: inspect (read the data), deduplicate (exact, hashed, or embedding-based), clean and filter (PII, toxicity, low quality, formatting junk), then format (tokenizer, chat template, loss mask).
• Manual inspection of a random sample remains the highest-value, lowest-prestige activity in the pipeline. Do it before and after every processing step.

Data curation

Which data you need

Different training stages need different data, and finetuning projects typically produce one of two kinds.

Instruction data for supervised finetuning is a set of (prompt, response) pairs, possibly multi-turn, where the response is what you want the model to say. Preference data for alignment methods such as RLHF and DPO is (prompt, preferred response, rejected response), telling the model which of two answers is better. Both are far smaller than pretraining corpora and far more sensitive to individual mistakes, because every example is a demonstration the model is explicitly asked to imitate or prefer.

Within instruction data, the book flags two refinements that often matter more than raw volume. First, if production conversations are multi-turn, train on multi-turn examples with the loss on the assistant turns; a model trained only on single exchanges handles follow-ups badly. Second, if you want the model to reason before answering, include the reasoning in the responses: chain-of-thought training data improves reasoning in a way that a "think step by step" instruction at inference cannot fully replace. Both decisions should be driven by what the production prompt looks like, including any system prompt, retrieved context or tool outputs, because the training examples must reproduce that shape.

Data quality

Quality is not one thing. The book lists six properties a high-quality example has, and each maps to a different kind of check.

Property	Meaning	How it fails
Relevant	The example is about the task and domain the model will serve	Generic chat data in a finetune for insurance claim triage
Aligned with task requirements	The response is the behaviour you actually want: correct, in the right tone, at the right length, with the right level of detail	Correct answers written in a style your product would never use
Consistent	Different examples (and annotators) agree on what a good response looks like	Two annotators score the same essay 3 and 8; one writes bullet points, one prose
Correctly formatted	Every example parses, uses the right template and has no stray artefacts	Trailing HTML, inconsistent JSON keys, missing end-of-turn tokens
Sufficiently unique	Few duplicates or near-duplicates	The same FAQ answer appears 400 times with trivial rewording
Compliant	Collected and used legally and within policy	Scraped content under a restrictive license; PII; a teacher model whose terms forbid training on its outputs

Consistency is the quietly hard one. Humans disagree about what a good response is, and a dataset that encodes several conflicting standards teaches the model to be inconsistent too. The fix is upstream: a written annotation guideline with examples of good and bad responses, used both to produce the data and to evaluate the model. The book makes the point that the guideline you use to judge model outputs during evaluation and the one you use to produce training data should be the same document.

The LIMA experiments (Zhou et al., 2023) are the standard illustration that quality and diversity beat volume. A 65B model finetuned on just 1,000 carefully selected examples produced responses that human raters preferred to or judged equal to those of much larger instruction-tuned systems in a large share of comparisons. Within the same study, filtered Stack Exchange data (high quality, diverse topics) beat wikiHow (high quality, one style) and unfiltered Stack Exchange (diverse, noisy): neither quality alone nor diversity alone was enough.

Quality and diversity both matter: filtered Stack Exchange data (high quality, diverse) beats wikiHow (high quality, not diverse) and unfiltered Stack Exchange (diverse, low quality)

Data coverage

Coverage is whether the dataset spans what the model will be asked to do. Think of the dimensions along which production requests vary and make sure each is represented in roughly production proportions: task types (summarize, extract, classify, answer, generate code), topics and domains, input lengths and output lengths, languages, user styles (terse versus verbose, typos, mixed formats), and edge cases such as empty inputs or adversarial prompts. A gap in coverage is a gap in capability, and because finetuning sharpens the model toward the data distribution, gaps can get worse rather than stay neutral.

Model providers publish their mixes, which are useful as reference points. Meta described Llama 3's pretraining data as roughly 50% general knowledge, 25% mathematics and reasoning, 17% code and 8% multilingual, and its post-training mix as dominated by general English with substantial reasoning, coding and exam-style data. The specific numbers matter less than the lesson: the mix was chosen deliberately and tuned against evaluations, not inherited from whatever was easiest to scrape.

Finetuning on more tasks improves held-out performance at every model size, with most of the gain in the first few hundred tasks

The instruction-tuning scaling work behind FLAN showed the same curve for task diversity: held-out performance improves as the number of distinct training tasks grows, at every model size, with most of the benefit arriving in the first few hundred tasks. For an application dataset the analogue is distinct kinds of request. Twenty examples each of fifty request types usually beats a thousand examples of one.

Data quantity

How much data you need depends on three things.

• The finetuning method. Parameter-efficient methods such as LoRA learn well from hundreds to a few thousand examples; full finetuning needs more to justify itself and to avoid over-fitting a huge parameter count to a tiny set.
• The task. Teaching a format or a tone takes less than teaching a new skill such as a domain-specific reasoning procedure.
• The base model. A stronger base needs fewer examples to adapt. OpenAI's finetuning guidance that 50 to 100 examples already produce visible changes applies to strong bases; a weaker model may need thousands to move.

With little data, the more capable base model wins by a wide margin; with lots of data, all models do well

The interplay between base model and data volume produces a practical rule. With little data, start from the most capable model you can afford: it will extract the most from each example. With a lot of data, smaller models catch up, and finetuning a small model becomes the cheaper route to the same quality. The book also notes the heuristic that more data lets you finetune a smaller model; equivalently, if you cannot get more data, buy capability instead.

Performance versus dataset size: steep gains at first, then diminishing returns

The small-data-first experiment. Before committing to an expensive data collection effort, measure the marginal value of data you already have.

Finetune on nested subsets (25%, 50%, 100%), evaluate each on the same held-out set, and look at the shape. A curve that is still climbing steeply says more examples will pay; a curve that has flattened says the bottleneck is elsewhere, usually coverage or quality, and ten times more of the same data will not fix it. Run this on a few hundred examples first; you learn almost as much about whether finetuning will work as you do about how much data it needs. Also check that the improvement comes from the data and not from the format: a finetune on fifty examples that mostly teaches the chat template can look like a large win on an evaluation set that penalized formatting failures.

Data acquisition and annotation

Data sources, roughly in order of value per example:

1. Application data. Logs of real user requests and the responses users accepted, corrected or rated. This matches the production distribution by construction and is the main reason the user-feedback design in the architecture chapter matters so much. Watch the consent and privacy terms under which it was collected.
2. Human-written data for your task. Expensive but highest quality when the guidelines are good. Domain tasks (legal, medical, financial) need domain experts, not generic crowdworkers, and the cost scales accordingly.
3. Public datasets. Hugging Face Hub, Kaggle, government open-data portals and research releases. Useful for coverage and for bootstrapping, less so for matching your exact distribution. Check licenses: many permit research but not commercial training.
4. Purchased or licensed data. Data vendors and content owners increasingly sell training rights; providers have struck licensing deals with publishers for exactly this reason.
5. Synthetic data. Generated by rules, simulators or models; the subject of the next section.

Annotation guidelines are the hard part of annotation. A guideline must define what the task is, what a correct response contains, what tone and length are expected, how to handle ambiguous or impossible inputs, and how to score or compare responses. Pilot it on a small batch, measure inter-annotator agreement, and revise before scaling; disagreement in the pilot is a guideline bug, not an annotator bug. Keep a gold set of adjudicated examples to check annotators against over time.

Human versus model annotation. Models can draft responses, rank candidates, or label attributes, and research from several labs has found model-written datasets approaching human quality for many tasks. The practical pattern is hybrid: a model generates or pre-labels, humans review a sample or adjudicate disagreements, and the review rate is set by how much the model's errors cost. Pure model annotation is cheapest and fastest; pure human annotation is the fallback when the task needs judgment the model lacks or when the data will be used to evaluate that same model (a model should not grade its own homework).

Data augmentation versus data synthesis

The book distinguishes two ways to make data. Augmentation starts from real examples and transforms them: flipping an image, paraphrasing a sentence, renaming variables in code. The output is derived from real data. Synthesis generates new examples that are not transformations of any particular real example: filling templates, running a simulator, prompting a model to write instructions. The line blurs in practice (is a back-translated sentence augmented or synthesized?), but the distinction is useful because augmentation preserves the real data's distribution while synthesis can extend or distort it.

Data synthesis

Why synthesize

Goal	What synthesis buys	Example
Quantity	More examples than you can collect or afford to annotate	Expanding 500 support tickets into 10,000 varied training conversations
Coverage	Examples for rare, dangerous or not-yet-seen cases	Edge-case driving scenarios; low-resource programming languages; adversarial prompts
Quality	Cleaner or more consistent examples than raw data	Rewriting messy human answers to one standard; generating code that is verified by tests
Privacy	Data with the statistical shape of sensitive records but none of the real people	Synthetic patient notes for training a clinical summarizer
Distillation	A small model learns from a large model's outputs	Frontier-model answers used to finetune a 7B model

Synthetic data can also be cheaper than human data by orders of magnitude and arrives in hours, which changes the economics of iteration: you can regenerate a dataset after changing the guideline rather than re-hire annotators.

Traditional synthesis techniques

Rule-based synthesis uses templates and transformations. Templates with slots ("Convert {amount} {from_currency} to {to_currency}") fill from value lists to produce arbitrarily many examples; they are deterministic, inspectable and cheap, and they produce data that is also predictable, so they are best for format and robustness coverage rather than for teaching skills. Transformations include synonym replacement, word deletion and reordering for text; variable renaming, reformatting, comment stripping and dead-code insertion for code; crops, flips and colour jitter for images. The constraint is that the transformation must preserve the label: renaming a variable keeps code correct, but deleting a random word can flip a sentence's meaning.

Perturbation adds controlled noise to make the model robust: typos, casing changes, unusual whitespace, distractor options in multiple-choice questions. Counterfactual augmentation swaps protected attributes (names, pronouns, nationalities) while holding everything else fixed, so the model learns the attribute is irrelevant; it is a standard tool for reducing bias in classifiers.

Simulation generates data by running a system rather than editing text. Self-driving companies synthesize rare traffic scenarios in simulators because collecting them on the road is slow and dangerous. Robotics trains in physics simulators and transfers to hardware. Game-playing agents such as AlphaZero generate their entire training set by self-play, with no human data at all. For language models, simulation shows up as simulated tool environments: a fake API that returns plausible responses lets you generate tool-use trajectories without hitting a real service, which is how toolbench-style datasets are built.

AI-powered synthesis

A capable language model is itself a data generator, and most post-training datasets since 2023 have been built at least partly this way.

Examples of synthesized training data: a retrieval-grounded answer with citations and a multi-turn coding conversation

Paraphrasing and translation. Ask a model to rewrite an example several ways and you get more coverage of surface forms for the same label. Back-translation (translate to another language and back) is the classic version. MetaMath applied rephrasing to maths problems (rewording questions, rewriting them backwards so a known quantity becomes the unknown) and improved reasoning accuracy substantially. The Llama 3 team translated Python examples into less common programming languages (PHP, TypeScript and others), generating code data for languages where little high-quality data exists, and generated natural-language explanations of code snippets to build documentation-style examples.

Simulated conversations. Two model instances, one playing the user and one the assistant, can generate multi-turn dialogues at scale. UltraChat built a large dialogue corpus this way, seeding the "user" with topics and personas to keep the conversations diverse. The risk is that the simulated user is too polite, too clear and too consistent, which real users are not; injecting realistic noise into the user side is part of the craft.

Instruction data synthesis. The landmark recipe is Self-Instruct (Wang et al., 2022): start from a small seed pool of human-written tasks, prompt a model with a few seeds to write new instructions, generate inputs and outputs for them, filter, and add the survivors to the pool for the next round. Stanford's Alpaca applied it with 175 seed tasks and OpenAI's text-davinci-003 to produce 52,000 instruction examples for a few hundred dollars, then finetuned LLaMA 7B into a usable assistant. Alpaca's filters are a good template for cheap quality control: drop new instructions whose ROUGE-L similarity to any existing instruction exceeds 0.7 (diversity), drop instructions containing keywords the model cannot act on such as "image" or "graph", and drop those that are too short or too long.

A seed task and a generated task from the Alpaca pipeline

Evol-Instruct (the WizardLM recipe) addresses Self-Instruct's tendency to produce easy instructions. Starting from a simple instruction, it asks the model to evolve it along specified axes: add constraints, deepen the question, concretize it, require more reasoning steps, or broaden to a related but harder topic. Several rounds of evolution produce instructions far harder than a model would write unprompted. Instruction backtranslation (Li et al., 2023) runs the pipeline in reverse: take high-quality human text from the web as the responses and ask the model to write the instruction each one would answer, then have the model score and keep only the best pairs. This exploits the fact that good answers already exist in enormous quantity; what is missing is the question.

Synthetic preference data. Preference pairs are expensive to collect from humans because each requires reading two responses. The synthetic route generates several candidate responses to a prompt (from different models, temperatures or prompts) and has a judge model rank them or score them against a rubric; the top and bottom become the chosen and rejected pair. UltraFeedback and many open preference datasets were built this way, and the judge's biases (length, style, position) become the dataset's biases unless deliberately controlled.

Long-context data. Long documents with useful questions about them are rare. The Llama 3 team chunked long documents into pieces, generated questions and answers per chunk and per document, and assembled them into training examples that exercise retrieval over a long context. Summarization of hierarchically chunked documents and multi-hop questions that span chunks are built the same way.

The Llama 3 coding data pipeline

The Llama 3 report describes the most detailed public pipeline for synthetic code data, which produced on the order of 2.7 million examples. It combines generation with objective verification and iterative correction.

1. Problem generation. Sample random code snippets and topics and ask the model to write programming problems inspired by them, across a wide range of difficulty and domain.
2. Solution generation. Ask the model to solve each problem, explaining its reasoning in comments and following good coding practice. Prompting for explanations improved solution quality.
3. Correctness analysis. Run static checks (the code parses, passes a linter) and dynamic checks (the model writes unit tests, the solution is executed against them in a sandbox).
4. Error feedback and self-correction. Failures are not simply discarded. The errors are fed back and the model is asked to revise; a meaningful fraction of initially wrong solutions became correct after revision.
5. Iterate. Finetune on the verified data, then use the improved model to generate the next round.

The transferable lesson is that generation is cheap and verification is what makes the data worth training on. Wherever an objective check exists (code executes, maths answer matches, SQL returns the right rows, JSON validates against a schema), build it into the pipeline and let the model try again when it fails.

Verifying synthetic data

Method	Mechanism	Best for	Limits
Functional correctness	Execute code, check answers, validate schemas, run proof checkers	Code, maths, structured outputs, tool calls	Only where a ground truth or executable check exists
AI judge	A strong model scores or ranks examples against a rubric	Open-ended text: helpfulness, tone, faithfulness	Inherits the judge's biases; costs a model call per example
Heuristics	Length bounds, repetition, language ID, perplexity thresholds, keyword filters, near-duplicate similarity	Removing obvious junk cheaply at scale	Catches surface problems only
Consistency checks	Compare multiple generations or multiple judges; keep agreement	Flagging uncertain examples	Shared systematic errors pass
Human sampling	Experts review a random sample and the tails	Calibrating the automated filters; catching what they miss	Does not scale; use to set thresholds

Layer them: heuristics first to cut volume, functional checks where they exist, a judge on the remainder, and a human sample to measure how often each layer is wrong. Record the pass rate of every filter; a filter that removes 60% of generations is telling you about the generator.

Limitations of AI-generated data

Quality control at scale. Model errors are fluent. A wrong fact in a well-structured paragraph passes every heuristic and most judges, and when thousands of such examples enter the training set the student learns to be confidently wrong. The Llama 3 team's insistence on execution-based verification exists because judges are not enough for correctness.

Superficial imitation. Gudibande et al. (2023), in "The False Promise of Imitating Proprietary LLMs", finetuned open models on growing amounts of ChatGPT output. Human raters increasingly preferred the imitators, but on targeted benchmarks the imitators barely improved and often hallucinated more. The students had learned the teacher's style (structure, hedging, length) without its knowledge, and raters mistook the style for competence. Knowledge comes mostly from pretraining; a few thousand imitation examples adjust presentation. This is why distillation works best for tasks the student is already capable of and why benchmark evaluation must accompany human preference.

GPT-4 responses are longer on average than GPT-3 responses, so a model trained on them learns to write longer answers

The style transfer is measurable. Data generated with GPT-4 is longer on average than data generated with earlier models, so students trained on it become more verbose whether or not verbosity helps. And more capable generators are not automatically more diverse: in one comparison, GPT-4-generated instructions had fewer unique verb-noun pairs than GPT-3-generated ones (5,229 versus 6,133). Stronger models converge on their favourite phrasings; diversity has to be engineered in through seeds, personas and topic sampling.

Top verb-noun pairs in GPT-4 versus GPT-3 generated instructions: GPT-4 data is less diverse (5,229 unique pairs versus 6,133)

Model collapse. Shumailov et al. showed that training models recursively on their own generations degrades them generation by generation: the tails of the distribution disappear first (rare facts, unusual phrasings, minority styles), and the output distribution narrows toward a bland mode. The mechanism is statistical: each generation samples from the previous model, sampling loses low-probability events, and the next model never sees them. Follow-up work (Gerstgrasser et al., 2024) found that collapse is largely avoided when synthetic data accumulates alongside the original human data rather than replacing it, which is the practical defence: keep the human data, cap the synthetic fraction, and refresh from real sources. The ecosystem-level concern is that web-scraped pretraining data now contains a growing share of model output that no one labelled as such.

Obscured data lineage. When a model generates your training data, you cannot trace which of its training examples influenced which output. That makes it hard to answer whether the synthetic data carries copyrighted text, memorized PII, benchmark test items (contamination) or a license the teacher's terms forbid. Several providers prohibit using their outputs to train competing models; some open models (Llama 3.1, Nemotron-4 340B) are explicitly licensed for synthetic data generation. Document the generator, its version, the prompts and the seeds for every synthetic dataset you ship.

Model distillation

Distillation trains a small student to reproduce the behaviour of a large teacher. In Hinton et al.'s original formulation (2015) the student matches the teacher's full output distribution (soft labels), which carries more information than hard labels because it encodes which wrong answers the teacher considered plausible. DistilBERT applied this to BERT and kept about 97% of its language-understanding performance with 40% fewer parameters and 60% faster inference.

With API-only teachers you cannot see the distribution, so modern LLM distillation is usually black-box: the teacher generates responses, the student is finetuned on them with ordinary supervised learning. Alpaca is the canonical example. NVIDIA's Nemotron-4 340B pushed the idea furthest by building over 98% of its alignment data synthetically, with a reward model of its own in the loop to filter it, and shipping the model under a license that invites others to do the same.

Distillation is the most common reason application teams synthesize data, and its two caveats are the ones above. It transfers style more readily than knowledge, so pick a student whose pretraining already covers the domain, and it is bounded by the teacher's terms of service, so check them before generating.

Data processing

The processing pipeline turns collected and synthesized examples into a training set. The order is deliberate: inspection informs every later choice, deduplication should happen before filtering (so filter statistics are not skewed by repeats) and before splitting (so the validation set does not contain copies of training examples), and formatting comes last because it depends on the exact model being trained.

Inspect the data

Look at the data before doing anything to it. Compute distributions: input and output lengths in tokens, language, task type (a quick classifier or keyword rules will do), topic clusters from embeddings, source, date. Compare them to what you know about production traffic. Then read: a random sample of at least a hundred examples, plus the tails (shortest, longest, highest and lowest judge scores, most and least common topics). Reading finds the problems that statistics hide: the annotator who answered every question with a bulleted list, the template that leaked into 3% of examples, the responses that quietly refuse, the system prompt that is missing from half the conversations.

"Manual inspection of data has probably the highest value-to-prestige ratio of any activity in machine learning." (Greg Brockman)

Make inspection a step in the pipeline, not a one-off. Re-sample after deduplication, after filtering and after formatting, because each step can introduce a new systematic problem; the formatting step in particular is where a wrong template or a mis-set loss mask ruins a run.

Deduplicate

Duplicates distort training in three ways: they over-weight whatever is repeated (the model learns the repeated examples disproportionately well, including their mistakes), they encourage memorization and regurgitation, and they leak across the train/validation split so your evaluation lies to you. Lee et al. (2021) found that deduplicating pretraining data improved models while cutting training cost. Duplication also occurs at several granularities: whole examples, passages inside long documents, and the same example appearing in two source datasets.

Method	How it works	Catches	Cost	Notes
Exact match	Hash each normalized example; drop collisions	Verbatim copies	Linear	Normalize whitespace, case and punctuation first or you will miss trivial variants
Pairwise comparison	Compute a similarity (edit distance, n-gram Jaccard) between every pair	Near-duplicates with high recall	Quadratic	Only feasible for small datasets or within pre-clustered buckets
Hashing (MinHash + LSH)	Shingle each example into n-grams, compute MinHash signatures approximating Jaccard similarity, bucket with locality-sensitive hashing so only likely pairs are compared	Near-duplicates at web scale	Near linear	The standard for large text corpora; tune shingle size and similarity threshold
Bloom filters	Probabilistic set membership for seen n-grams	Repeated passages within and across documents	Linear, tiny memory	False positives possible, no false negatives
Dimensionality reduction (embeddings)	Embed each example, reduce dimension, cluster or nearest-neighbour search; drop examples within a semantic distance	Paraphrases and semantic duplicates	GPU for embeddings, then ANN search	SemDeDup-style; threshold-sensitive, can remove legitimate variety

For instruction data, a common recipe is exact dedup on normalized text, MinHash near-dedup on prompts (two different prompts with the same response are usually fine; two near-identical prompts with different responses are a consistency problem to inspect), and embedding-based dedup only when the data is small enough to afford it and diverse enough that paraphrase duplicates are a real concern. Always dedup across the train/validation boundary.

Clean and filter

Cleaning removes or repairs content inside examples; filtering removes whole examples. Both must be done with the production distribution in mind, because every filter you add changes what the model learns.

Formatting junk. Strip HTML tags, navigation boilerplate, markdown artefacts from a different renderer, control characters, broken encodings and stray special tokens from other models' templates. Be careful not to strip structure the model should learn: tables, code fences and headings in responses are content, not junk.

PII. Detect and remove or pseudonymize names, email addresses, phone numbers, government IDs, account numbers, addresses and free-text health or financial details. Tools such as Microsoft Presidio combine regex, checksums and NER models; domain-specific identifiers need custom patterns. Decide between removal, masking and consistent pseudonymization (replace each real name with the same fake name throughout a conversation) based on whether the model needs the entity to stay coherent.

Toxicity and policy violations. Classifiers (Perspective API, open toxicity models, your own) flag hate speech, harassment, sexual content and other categories; the threshold depends on the product and on whether the example is teaching the model to produce or to refuse such content. Safety training data deliberately includes harmful prompts paired with good refusals, so do not filter prompts by toxicity without looking at the responses.

Low quality. Heuristics: responses that are too short to be useful or too long for the task, high repetition ratios, mismatched language between prompt and response, truncated outputs, responses that do not address the prompt. Perplexity under a reference model flags statistically odd text (both garbage and, unfortunately, unusual but valid text). Model-based quality classifiers, trained on a small labelled set of good and bad examples, were central to Llama 3's data filtering and scale better than hand rules.

Compliance. Drop content whose license or terms do not permit training, and keep provenance metadata so you can drop it later if terms change.

Over-filtering is the failure mode to watch. Aggressive filters remove exactly the unusual examples (dialects, minority topics, long technical answers, legitimate discussion of sensitive subjects) that give a dataset its coverage, and the loss is invisible unless you measure the distribution before and after. Log what each filter removed, sample from the rejects, and reconsider any filter whose rejects look fine.

Format the data

Formatting converts cleaned examples into the exact token sequences the model will be trained on. Three mistakes here each silently waste a training run.

Tokenizer. Use the tokenizer that ships with the base model, at the same version. A different tokenizer, or a mismatched special-token configuration, produces token IDs the model has never seen. Verify by decoding a few tokenized examples back to text and checking for unexpected unknown-token IDs.

Chat template. Instruction-tuned models were trained with a specific template of special tokens marking roles and turn boundaries, and finetuning data must use the same one or the model will misread where the user stopped and the assistant should start. Templates differ sharply across families. Llama 2 Chat wraps the user turn in [INST] ... [/INST] and the system prompt in <<SYS>> ... <</SYS>> inside the first instruction, while Llama 3 uses header tokens and an end-of-turn token:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a concise assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

Summarize the ticket below in one sentence.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Customer reports duplicate charges on the 3rd and asks for a refund.<|eot_id|>

A ChatML-style model instead uses <|im_start|>role and <|im_end|>. Using the Llama 2 template on a Llama 3 model trains the model to emit literal [/INST] strings and to ignore <|eot_id|>; at inference it rambles past the end of its answer. The safe path is to use the template stored in the model's tokenizer configuration (Hugging Face exposes it as apply_chat_template) and never to hand-write one.

Instruction/response structure and the loss mask. Decide what the model is trained to predict. For single-turn data, the loss is usually computed on response tokens only; for multi-turn, on every assistant turn. Mark the prompt and system tokens so they are excluded from the loss (the prompt loss weight discussed in the finetuning chapter). Confirm the mask by printing which tokens carry loss for a few examples. Also include in the training examples everything the production prompt contains: if production has a system prompt, retrieved passages or tool results, the training data needs the same elements in the same positions, or the finetuned behaviour will not transfer.

Packing and truncation. Short examples are often packed several to a sequence to fill the context window; the attention mask must prevent examples from attending to each other, and position IDs should restart per example. Examples longer than the context are truncated; truncate from the prompt side when the response is what matters, and check how many examples are affected, because a dataset where 20% of responses are cut off trains a model that stops mid-sentence.

Finish with a dry run: take ten formatted examples, decode them, read them, and confirm the template, the roles, the special tokens and the loss mask are all what you intended. Ten minutes here saves days of GPU time.

Design trade-offs

Choosing a data source

Dimension	Application logs	Human annotation	Public datasets	Synthetic (model-generated)
Match to production distribution	Highest	High if guidelines are good	Low to medium	Medium; depends on seeds and prompts
Quality ceiling	Variable; needs filtering	Highest for expert tasks	Variable	Bounded by teacher and verification
Cost per example	Near zero to collect, real cost to label	Highest	Free to cheap	Cents
Time to 10k examples	Depends on traffic	Weeks	Hours	Hours
Coverage of rare cases	Poor (rare is rare)	Good if targeted	Random	Excellent if targeted
Legal and privacy risk	Consent and PII	Low	License terms	Teacher terms, lineage
Best use	Core training signal and evaluation	Gold standard and guideline development	Bootstrapping and coverage	Scale, edge cases, distillation

Choosing a verification strategy

Pick the strongest check the task admits: execution or exact-match for code, maths, SQL and structured outputs; schema validation plus an AI judge for formatted text; a rubric-driven judge plus length and style controls for open text; and a human sample in every case to measure the automated layers' error rates. If no check beats a coin flip, the task is not ready for synthetic data.

Interview angle

• You have 300 labelled examples and a budget to collect more. How do you decide whether to spend it? Run the small-data experiment: finetune on 25%, 50% and 100% and evaluate each on a fixed held-out set. A rising curve says more similar data will help; a flat curve says the limit is coverage, quality or the method, and more of the same is wasted. Also compare against a stronger base model prompted or finetuned on the same 300, since capability can substitute for data.

• How would you build a synthetic instruction dataset and keep it from being junk? Self-Instruct from a seed pool with diverse personas and topics, Evol-Instruct rounds to raise difficulty, and filters: ROUGE-L against existing instructions for diversity, length and keyword heuristics, a judge rubric for helpfulness, and execution checks wherever the task is verifiable. Track filter pass rates, read a sample of rejects and keeps, and benchmark the finetuned model rather than trusting judge scores.

• A team distilled GPT-4 answers into a 7B model; human raters love it but accuracy on internal tests did not move. Explain. Superficial imitation: the student learned the teacher's format and tone, which raters reward, without the knowledge, which comes from pretraining. Fixes: distil onto a base already competent in the domain, add verified or retrieval-grounded examples, evaluate with task benchmarks alongside preference ratings, and use RAG for the knowledge part.

• What is model collapse and how do you avoid it when using synthetic data? Recursive training on model outputs loses the distribution's tails and narrows toward the mode, degrading each generation. Avoid replacing real data with synthetic data: accumulate synthetic alongside human data, cap the synthetic fraction, verify synthetic examples, keep refreshing from real sources, and track output diversity across model versions.

• Design a deduplication pipeline for 50 million documents. Normalize text; exact dedup by hash; near-dedup with MinHash over word shingles and LSH bucketing at a Jaccard threshold around 0.8, comparing only within buckets; optionally Bloom filters for repeated passages inside documents; embedding-based semantic dedup only on a smaller high-value subset. Deduplicate before splitting so validation never contains training copies.

• Why does the chat template matter so much, and how do you check it? The base model learned a specific set of special tokens for roles and turn boundaries. Training with a different template teaches it to emit or ignore the wrong tokens, producing a model that rambles or misreads turns. Use the template from the model's tokenizer config, decode a few formatted examples, and print which token positions carry loss to verify the mask.

• What makes a good annotation guideline and why does it double as your evaluation rubric? It defines the task, what a correct and well-formed response contains, tone and length, how to treat ambiguous or impossible inputs, and how to compare responses, with worked examples of good and bad. Pilot it, measure inter-annotator agreement, revise. The same definition of "good" should produce the training data and grade the model, otherwise you train toward one standard and measure against another.

• When should you prefer a rule-based or simulation-based generator over a language model? When you need determinism, guaranteed label correctness or exhaustive coverage of a structured space: format robustness tests, counterfactual bias augmentation, tool-call trajectories against a mock API, rare scenarios in a simulator. Model generation wins for open-ended language where templates would be obvious and brittle.

Pitfalls and misconceptions

• Training on examples that do not look like production prompts. If production includes a system prompt, retrieved context or tool outputs and the training data does not, the finetuned behaviour will not transfer. Reproduce the full prompt shape.
• "More data is always better." Past the knee of the curve, more of the same data adds nothing; duplicates and low-quality examples actively hurt. Fix coverage and quality before volume, and measure the marginal value with subset experiments.
• Trusting an AI judge as ground truth. Judges prefer long, confident, well-formatted answers and share the generator's blind spots. Use execution checks where possible and a human sample to calibrate the judge.
• Filtering prompts by toxicity without reading responses. Safety data deliberately pairs harmful prompts with good refusals; a prompt-side filter deletes the examples that teach refusal.
• Deduplicating after splitting. Copies across train and validation inflate every metric. Dedup first, split second.
• Assuming a more capable generator yields more diverse data. Stronger models converge on favourite phrasings; GPT-4-generated instructions were measurably less diverse than GPT-3's in one study. Engineer diversity through seeds, personas and topic sampling.
• Hand-writing the chat template. Off-by-one token mistakes are invisible in text and fatal in training. Use the tokenizer's stored template and verify by decoding.
• Ignoring lineage and license. Teacher terms, scraped-content licenses and PII all follow synthetic data downstream. Record generator, version, prompts and seeds for every dataset.

Key terms

Term	Meaning
Instruction data	(prompt, response) pairs, often multi-turn, used for supervised finetuning
Preference data	(prompt, chosen, rejected) triples used for RLHF, DPO and similar alignment methods
Data coverage	How completely a dataset spans the tasks, topics, lengths, languages and styles seen in production
Annotation guideline	The written definition of a good response used to produce training data and to evaluate the model
Inter-annotator agreement	How consistently different annotators label the same item; low agreement signals an ambiguous guideline
Data augmentation	Creating new examples by transforming real ones (paraphrase, flip, rename) while preserving the label
Data synthesis	Generating new examples by rules, simulation or a model rather than transforming real data
Counterfactual augmentation	Swapping a protected attribute while holding the rest fixed so the model learns it is irrelevant
Self-play	An agent generating its own training data by interacting with copies of itself
Self-Instruct	Bootstrapping instruction data from a seed pool by having a model write new tasks, inputs and outputs
Evol-Instruct	Iteratively rewriting instructions to be harder (more constraints, deeper reasoning, broader scope)
Instruction backtranslation	Generating the instruction for existing high-quality text, then self-curating the best pairs
Back-translation	Translating text to another language and back to produce paraphrases
AI judge	A model used to score or rank candidate outputs against a rubric
Functional verification	Checking generated data by executing it or comparing to a known answer (tests, exact match, schema)
Superficial imitation	A student learning a teacher's style without its knowledge, so preference ratings rise while accuracy does not
Model collapse	Progressive loss of distributional diversity when models are trained recursively on their own outputs
Data lineage	The traceable provenance of each example: source, license, generator, prompts
Distillation	Training a small student to reproduce a large teacher's outputs (soft labels) or responses (black-box)
MinHash / LSH	Hash-based approximation of set similarity and bucketing that finds near-duplicates in near-linear time
Semantic deduplication	Removing examples whose embeddings fall within a distance threshold of another's
PII	Personally identifiable information, to be removed or pseudonymized before training
Chat template	The model-specific arrangement of special tokens marking roles and turn boundaries
Loss mask	The per-token flags that decide which tokens (usually assistant responses) contribute to the training loss
Sequence packing	Concatenating several short examples into one training sequence with attention masks that keep them separate

Further reading

• AI Engineering, Chapter 8 (Dataset Engineering); companion material at https://github.com/chiphuyen/aie-book
• Wang et al., "Self-Instruct: Aligning Language Models with Self-Generated Instructions" (2022): https://arxiv.org/abs/2212.10560
• Taori et al., "Alpaca: A Strong, Replicable Instruction-Following Model" (Stanford CRFM, 2023): https://crfm.stanford.edu/2023/03/13/alpaca.html
• Xu et al., "WizardLM: Empowering Large Language Models to Follow Complex Instructions" (Evol-Instruct, 2023): https://arxiv.org/abs/2304.12244
• Li et al., "Self-Alignment with Instruction Backtranslation" (2023): https://arxiv.org/abs/2308.06259
• Zhou et al., "LIMA: Less Is More for Alignment" (2023): https://arxiv.org/abs/2305.11206
• Gudibande et al., "The False Promise of Imitating Proprietary LLMs" (2023): https://arxiv.org/abs/2305.15717
• Shumailov et al., "The Curse of Recursion: Training on Generated Data Makes Models Forget" (2023): https://arxiv.org/abs/2305.17493
• Gerstgrasser et al., "Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data" (2024): https://arxiv.org/abs/2404.01413
• Grattafiori et al., "The Llama 3 Herd of Models" (2024), sections on post-training data and synthetic code data: https://arxiv.org/abs/2407.21783
• Lee et al., "Deduplicating Training Data Makes Language Models Better" (2021): https://arxiv.org/abs/2107.06499
• Abbas et al., "SemDeDup: Data-efficient learning at web-scale through semantic deduplication" (2023): https://arxiv.org/abs/2303.09540
• Hinton, Vinyals and Dean, "Distilling the Knowledge in a Neural Network" (2015): https://arxiv.org/abs/1503.02531
• Sanh et al., "DistilBERT, a distilled version of BERT" (2019): https://arxiv.org/abs/1910.01108
• Yu et al., "MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models" (2023): https://arxiv.org/abs/2309.12284
• Hugging Face, "Chat Templates" documentation: https://huggingface.co/docs/transformers/chat_templating
• Microsoft Presidio (PII detection and anonymization): https://microsoft.github.io/presidio/