Chapter 3 built the vocabulary of evaluation: exact versus subjective metrics, AI judges, comparative evaluation. This chapter turns that vocabulary into a working process. It answers three operational questions that every team shipping a foundation-model application has to answer. Which criteria actually matter for this application? Which model should we build on, and how do we decide between a hosted API and weights we run ourselves? And how do we build an evaluation pipeline that is reliable enough to make decisions with, rather than a demo that confirms what we already believed?

The reason to care is blunt. A team without a trustworthy evaluation pipeline cannot tell whether a prompt change, a model upgrade or a retrieval tweak made things better or worse. Every decision becomes a guess, and guesses compound. Huyen's framing is that the lack of a reliable evaluation pipeline is one of the biggest blockers to AI adoption, and the material below is essentially a playbook for removing that blocker.

In one minute

• Evaluate an application on five families of criteria: domain-specific capability, generation capability (fluency, coherence, faithfulness, relevance), factual consistency, safety and instruction-following, plus the operational pair cost and latency.
• Factual consistency comes in two flavours: local (is the output consistent with the supplied context?) and global (is it consistent with the world?). Verify it with self-verification (sample several answers and check agreement), knowledge-augmented verification (decompose into claims, look each one up) or textual-entailment classifiers.
• Model selection is a funnel: filter by hard attributes you cannot change (license, privacy, size for on-device), shortlist with public benchmarks, run your own evaluation on your own data, then monitor in production.
• "Open source" is a spectrum: open weight (weights only, often with a restrictive license) versus fully open (weights, data, code). Build-versus-buy is a trade of API cost against engineering cost, with data privacy, data lineage, functionality (logprobs, finetuning, structured outputs) and control on the scales.
• Public benchmarks saturate quickly and are vulnerable to data contamination; detect it with n-gram overlap or perplexity, and treat leaderboard scores as a shortlist tool, never as a decision.
• Model APIs change under you: continuous evaluation is the only way to detect drift before users do.
• Design the pipeline bottom-up: evaluate every component (and every turn, not just the task), write an evaluation guideline with a rubric and examples, tie scores to business metrics, slice the data, and size the eval set with bootstrapping.
• Any evaluation system is itself a system that can be wrong: measure its reliability and correlation with downstream outcomes, and iterate it as the application evolves.

Evaluation criteria

Every application has its own definition of "good". Before you can select a model or build a pipeline you need to write that definition down as a list of criteria, each with a method to measure it. The book groups criteria into capability criteria (what the model can do) and operational criteria (what it costs to run). Most applications end up with a handful from each.

Domain-specific capability

Domain-specific capability is whether the model knows the domain well enough to be useful in it: can it write idiomatic Python, read French legal contracts, interpret clinical notes? Public benchmarks exist for many domains (HumanEval and MBPP for code generation, MMLU subsets for professional exams, specialised sets like MedQA or LegalBench), and they are a reasonable first filter.

Two cautions apply. First, most domain benchmarks are multiple-choice, because multiple-choice is cheap to grade. That format tests recall and recognition well but says little about generation: a model that picks the right diagnosis from four options may still write an incoherent discharge summary. Second, public benchmarks are built from public text. A model trained on web crawls has seen the kind of material the benchmark is built from, so a strong score tells you more about coverage than about depth.

For your own evaluation, assemble 50 to a few hundred domain examples that are hard enough to separate candidates. Easy cases do not discriminate. Include the jargon, the edge cases, the reasoning patterns your experts actually use (differential diagnosis, statutory interpretation, the failure modes of a particular SQL dialect) and, if the application serves a region or community, its linguistic and cultural conventions.

Generation capability

Early NLG research evaluated generated text on fluency (is it grammatical, natural-sounding?) and coherence (does it hold together as a whole?). Modern foundation models rarely fail at either, so these criteria have faded into the background and attention has moved to three harder ones.

• Faithfulness (also called factual consistency or groundedness): does the output stay true to its source? A summary that adds a claim the article never made is unfaithful, however fluent.
• Relevance: does the output address what the user asked, at the right level of detail? A technically correct essay is irrelevant if the user wanted a yes-or-no answer.
• Safety: is the output free of toxic, biased or otherwise harmful content, including content that is harmless in general but inappropriate for this application?

The remaining criteria in this section are the ones that most frequently sink a product, so each gets its own treatment.

Factual consistency and hallucination

A hallucination is an output that is not grounded in fact. Measuring factual consistency is one of the most valuable things an evaluation pipeline can do, and also one of the hardest. The first thing to pin down is consistent with what.

• Local factual consistency judges the output against a supplied context: the retrieved documents in a RAG system, the article being summarised, the table being queried. The context is the ground truth, even if the context itself is wrong. This is the easier setting because the verifier only has to compare two texts.
• Global factual consistency judges the output against open knowledge: is "the Eiffel Tower is 330 metres tall" actually true? This requires an external source of truth, so it is harder, slower and less reliable.

Why do models hallucinate at all? The book collects two hypotheses. One is that the model cannot distinguish between what it was given and what it generated itself (a kind of self-delusion): once an incorrect token is in the context, the model treats it as fact and builds on it, which is why hallucinations often snowball into increasingly confident nonsense. The other is that hallucination is partly a side-effect of supervised finetuning: if annotators write answers that draw on knowledge the model does not have, the model learns that it is acceptable to produce confident answers whenever it does not know. Both hypotheses point the same way: give the model the facts it needs, and train or prompt it to say "I don't know".

Verifying factual consistency. Three families of techniques are in common use.

Technique	How it works	Where it fits	Weakness
Self-verification (SelfCheckGPT style)	Sample several responses to the same prompt; claims that recur across samples are likely grounded, claims that vary are likely hallucinated	No external knowledge source, cheap to set up	Costs N generations per check; a consistently wrong model passes
Knowledge-augmented verification (SAFE, FActScore style)	Decompose the output into atomic claims, retrieve evidence for each (search engine, knowledge base), have a model judge whether evidence supports the claim	Global factual consistency of long-form answers	Depends on retrieval quality and judge quality; slow
Textual entailment / NLI	Treat the context as premise and each output sentence as hypothesis; a classifier labels the pair entailment, contradiction or neutral	Local factual consistency, summarisation, RAG	Needs a capable entailment model; struggles with long contexts

Textual entailment deserves a closer look because it is the workhorse for RAG and summarisation. A sentence that is entailed by the context is supported, a sentence that contradicts it is a hallucination, and a neutral sentence is something the model added that the context neither confirms nor denies. Neutral is the hard case: in a factual summary it is usually a problem, in a creative assistant it is fine. Your guideline has to decide.

Accuracy on adversarial TruthfulQA questions: RLHF-trained models are markedly more truthful than their base models

The TruthfulQA benchmark is the standard public probe for truthfulness. Its questions are adversarial (common misconceptions, superstitions, conspiracy theories) and its original paper found that larger base models were less truthful, because they had learned the misconceptions better. The figure above, from the InstructGPT work, shows the counterweight: preference finetuning raises truthfulness substantially, which is why instruction-tuned models dominate this benchmark today.

Safety

Safety covers everything that could harm a user or embarrass the operator: toxic and hateful content, sexual content outside the intended use, instructions for violence, privacy leaks, and the quieter problem of bias. Bias has an explicit form (stereotyped statements) and an implicit form (systematically different treatment of different groups in otherwise neutral tasks, such as scoring essays or screening resumes). Evaluate both, and evaluate under adversarial inputs, because the baseline behaviour of a modern model is well-behaved and the risk is in what users can provoke.

Political and economic leanings of different foundation models on a political compass

The political compass figure above illustrates a subtler point. Models have measurable political and economic leanings, mostly a product of their training data and finetuning choices, and those leanings differ between models. For a consumer product this can be a brand risk; for a model that moderates content or drafts policy it is a correctness risk. If it matters for your application, measure it rather than assume neutrality.

Instruction-following capability

Most production applications wrap the model in a system prompt that imposes a format, a persona, a length limit or a set of rules. Instruction-following capability is how reliably the model honours those constraints, and it is distinct from domain capability: a model can know everything about tax law and still ignore "respond in JSON".

Two public benchmarks show the two ways to measure it.

• IFEval (Google, 2023) uses only verifiable instructions: constraints whose satisfaction can be checked by a program, such as "write at least 300 words", "mention the keyword X three times", "do not use commas", "end with the phrase Y". There are 25 instruction types; scoring is exact and automatic.
• INFOBench (2024) covers the open-ended kind, including content constraints ("only discuss climate change"), style, format and linguistic constraints. Each instruction is decomposed into yes/no questions and a judge (human or model) answers them, giving a decomposed-requirements-following ratio.

Verifiable instructions are the right starting point for your own eval set because they need no judge. Add judge-based checks for the constraints you cannot verify programmatically, and keep the two scores separate.

The ten most common instruction types in LMSYS conversations

Real traffic is dominated by a small number of instruction types, as the LMSYS data above shows. Your eval set should mirror the distribution of instructions your users actually send, weighted by how expensive a failure is, rather than mirroring a benchmark.

Roleplaying is a special case of instruction following that matters for companion, gaming and persona-driven products: can the model stay in character over a long conversation, and does its knowledge stay consistent with the character? Benchmarks such as RoleLLM and CharacterEval exist, but the practical test is still a long scripted conversation judged against a character sheet.

Cost and latency

Capability criteria decide whether a model can do the job; cost and latency decide whether you can afford to let it. The two are in tension with capability (bigger models are better, slower and dearer) and with each other (lower latency usually means more hardware per request).

Latency for an autoregressive model is not one number. Track at least three.

Metric	Definition	What it governs
Time to first token (TTFT)	Delay from request to the first output token; dominated by prefill of the prompt	Perceived responsiveness in streaming UIs; grows with prompt length
Time per output token (TPOT)	Average interval between consecutive output tokens during decoding	Whether text appears faster than a person reads (roughly 6 to 8 tokens per second is the comfortable floor)
Total latency	Time to the complete response, roughly TTFT + TPOT x output tokens, plus network and application overhead	Non-streaming use cases and anything downstream of the model

Report latency as percentiles (p50, p90, p99), never as a mean: a long-tail p99 is what users notice. Latency also depends on the prompt and the output length, so the same model has a different profile for a classification task than for an essay generator. Test with your prompts, not the vendor's.

Cost for an API model is per token, with output tokens typically priced three to four times higher than input tokens, so verbose responses and long chain-of-thought traces cost real money. Cost for a self-hosted model is mostly compute: the GPUs run whether or not requests arrive, so unit cost falls with utilisation. That asymmetry is the heart of the build-versus-buy decision in the next section.

A final rule: do not optimise cost or latency before you have established quality. Start with the strongest model that satisfies your hard constraints, prove the task is solvable, then work downward to cheaper and faster options with the eval pipeline as your safety net.

The model selection workflow

With criteria written down, you can select a model. There are hundreds of candidates and the list changes monthly, so you need a repeatable process rather than a one-off bake-off. The book's process is a funnel built on one distinction.

Hard attributes versus soft attributes

A hard attribute is a property of a model that you cannot change and that the model provider (or your policy) has fixed: its license and commercial terms, whether its training data is disclosed, whether it can be run on your hardware or on-device, whether the provider's data policy meets your privacy requirements, its context length, and the modalities it accepts. A hard attribute is a binary filter: a model either passes or it is out.

A soft attribute is a property you can improve with the techniques in the rest of the book: accuracy on your task, factual consistency, instruction following, toxicity rate, and to a lesser degree cost and latency (through quantisation, caching, batching and prompt compression). Soft attributes are what you optimise after filtering.

The distinction stops two common mistakes. One is wasting weeks evaluating a model that your legal team would never approve. The other is rejecting a model on a first-pass accuracy number that a few hours of prompt engineering would have fixed.

The four-step process

Model selection workflow: filter by hard attributes, use public benchmarks to shortlist, run task-specific evaluation, then monitor in production

1. Filter by hard attributes. Apply license, privacy, deployment and modality constraints. This also includes the build-versus-buy decision below, because "must run on-device" or "data cannot leave our VPC" eliminates every hosted API at a stroke.
2. Shortlist with public information. Use benchmarks and leaderboards to cut the survivors down to a handful. The goal is to spend your expensive evaluation budget on plausible candidates, not to pick a winner. The section on public benchmarks explains why these numbers cannot be trusted further than that.
3. Run your own evaluation. Evaluate the shortlist on your own prompts, your own data and your own criteria, with the pipeline described at the end of the chapter. Measure cost and latency under your traffic profile at the same time.
4. Monitor in production. The chosen model will change (API providers update silently), your traffic will change, and new models will appear. Production monitoring and user feedback feed back into step one on a schedule.

The funnel is cheap at the top and expensive at the bottom, which is the point: the costly step, evaluating on your own data, is only run on models that already pass every filter you can apply for free.

Build versus buy

The biggest hard-attribute decision is whether to call a hosted model API (OpenAI, Anthropic, Google, and the many providers that host open models) or to self-host a model whose weights you have. Before weighing the trade-offs, the vocabulary needs tightening, because "open source" is used loosely.

Open source, open weight and proprietary

• A proprietary model is available only behind an API. You never see the weights.
• An open-weight model ships its weights, so you can run, inspect and finetune it, but not its training data or full training recipe. Most "open" models (the Llama family, Mistral's releases, Qwen) are open-weight, and several carry licenses with usage limits (user-count thresholds, restrictions on training other models) that disqualify them from the strict definition of open source.
• A fully open model releases weights, training data and code (OLMo is the usual example). These are rare because training data carries legal and competitive risk.

Open-weight models are the practical middle ground. You get almost everything you want from openness (control, customisation, on-device deployment) and almost none of the transparency about data that would let you audit for contamination, bias or copyright exposure.

The dimensions of the decision

Data privacy. With an API, your prompts, which may contain customer data, source code or trade secrets, leave your infrastructure. Providers publish retention and training policies, and enterprise tiers typically promise zero retention, but the risk is real: the widely reported Samsung incident, in which employees pasted confidential source code into ChatGPT, is the canonical example. Regulated industries, data-residency laws and contractual obligations often force self-hosting, or at least a provider that runs in your cloud region.

Data lineage and copyright. You rarely know what a model was trained on, and that creates two exposures. The model may regurgitate copyrighted material into your product, and the model itself may become legally encumbered if a lawsuit against its creator succeeds (Getty Images versus Stability AI is the long-running example). Some API providers offer indemnification against copyright claims arising from their outputs; most open-weight licenses offer nothing. Where lineage matters, prefer models whose creators document their data or indemnify you.

Performance. Frontier proprietary models have historically led on the hardest benchmarks, but the gap has repeatedly closed within months as open-weight releases catch up, and for most non-frontier tasks it has effectively vanished, especially after finetuning on domain data.

Closed-source versus open-weight models on 5-shot MMLU over time: the gap keeps closing

Functionality. An API bundles the model with an inference service: scaling, batching, structured-output enforcement, function calling, caching, safety filters. Rebuilding that service is a substantial engineering project (Chapter 9). On the other hand, APIs often withhold things you get for free from your own weights: full logprobs (several providers expose none, or only a few top candidates, because logprobs make distillation easier), the ability to finetune, and access to internal activations. If your evaluation or classification pipeline depends on logprobs, that is a hard attribute.

An inference service wraps a model behind an API that users send prompts to and receive responses from

API cost versus engineering cost. An API has near-zero fixed cost and a linear variable cost. Self-hosting has a large fixed cost (GPUs, an inference engine such as vLLM or TensorRT-LLM, on-call staff who understand them) and a low marginal cost. The break-even point depends on volume, utilisation and how much the engineering time would otherwise be worth; teams routinely underestimate the last term. A common trajectory is to prototype and launch on an API, then move the highest-volume, most stable workloads to self-hosted open-weight models once the quality bar is established and the traffic justifies it.

Control, access and transparency. With an API you accept the provider's rate limits, their deprecation schedule, their content policy (which can refuse legitimate requests in your domain) and their silent model updates. With your own weights, the model only changes when you change it, and you can inspect it. The enterprise surveys the book cites put control and customisability ahead of cost as reasons to adopt open models.

Why enterprises care about open models: control, customizability and cost

On-device deployment. If the product must work offline, on a phone, in a vehicle or in a factory with no reliable connectivity, a hosted API is not an option. Only open-weight models, usually small and quantised, can run there, and model size becomes a hard attribute: a model with N billion parameters needs roughly 2N GB in 16-bit precision and about N/2 GB at 4-bit, before the KV cache.

Dimension	Model API (proprietary or hosted)	Self-hosted open-weight
Data leaves your infrastructure	Yes, subject to provider policy	No
Training data transparency	None	Rarely more than a model card
Copyright indemnification	Sometimes offered	Almost never
Frontier performance	Leads at the top end	Catches up within months; wins after domain finetuning
Logprobs, finetuning, internals	Limited or absent	Full access
Inference service (scaling, structured output, tools)	Included	You build or adopt (vLLM, TGI, TensorRT-LLM)
Cost shape	Linear per token, zero fixed	High fixed, low marginal
Control over model changes	Provider updates silently	Changes only when you change it
On-device	Not possible	Possible with small, quantised models
Time to first version	Days	Weeks to months

Navigating public benchmarks

Public benchmarks are how step two of the funnel works, and they are also where the most expensive mistakes are made. Understanding how they are built, aggregated and corrupted is what lets you use them without being misled.

Benchmark selection and aggregation

There are hundreds of public benchmarks, each measuring a narrow thing: a few hundred to a few thousand items on knowledge (MMLU, ARC), commonsense (HellaSwag, WinoGrande), math (GSM-8K, MATH), code (HumanEval, MBPP), truthfulness (TruthfulQA) or instruction following (IFEval). No one of them captures your application, so you have to choose a set and combine the scores.

Selection should follow your criteria list. If the product is a coding assistant, HumanEval-style and repository-level benchmarks matter and HellaSwag does not. If it summarises contracts, look for faithfulness and long-context benchmarks. Public benchmarks also saturate: once leading models score above 90 percent, a benchmark can no longer rank them, which is why MMLU gave way to MMLU-Pro and GPQA and GSM-8K to harder math sets.

Aggregation is where leaderboards hide their assumptions. The Hugging Face Open LLM Leaderboard averaged scores across its benchmarks, treating a point on TruthfulQA as worth a point on MMLU. HELM reports a mean win rate: for each benchmark, the fraction of other models a given model beats, averaged across benchmarks, which is robust to differing scales but harder to interpret. Chatbot Arena avoids static benchmarks altogether and ranks models by crowd-sourced pairwise preferences, using a Bradley-Terry model (an Elo-style rating) on hundreds of thousands of human votes.

One analysis the book highlights: benchmarks on the same leaderboard are not independent. Across the thousands of models on the Open LLM Leaderboard, the knowledge and commonsense sets (MMLU, ARC, HellaSwag, WinoGrande) correlated strongly with each other, while TruthfulQA correlated weakly with the rest. Averaging highly correlated benchmarks double-counts one skill and dilutes the uncorrelated one. If truthfulness is your bottleneck, a leaderboard average will barely register improvements in it.

The practical response is a custom leaderboard: pick the public benchmarks that map to your criteria, weight them by business importance, and rank the candidates yourself. It takes an afternoon with an evaluation harness (EleutherAI's lm-evaluation-harness and HELM both support hundreds of benchmarks) and produces a shortlist that is actually about your problem.

Leaderboards and model drift

Even a perfectly chosen leaderboard describes a model at one moment. Hosted models change. In 2023 a Stanford and Berkeley study compared the March and June versions of GPT-3.5 and GPT-4 on several tasks and found large swings in both directions: GPT-4's accuracy on a prime-number identification task fell from 84 percent to 51 percent in the revised version of the study (the first version reported an even steeper drop), while some other tasks improved. Whether the model got "worse" depends on which task you measure, but the model your application depends on was no longer the model you evaluated.

Changes in GPT-3.5 and GPT-4 behaviour between March and June 2023 on four tasks

This is model drift, and it has one defence: continuous evaluation. Pin model versions where the provider allows it, run your evaluation suite on a schedule and on every provider announcement, and alert on regressions. The same applies when a provider retires a version, which happens on a timescale of months.

Data contamination and how to detect it

Data contamination is when the benchmark's test items (or near-paraphrases of them) were in the model's training data. The model then scores well by recall, not capability. Contamination is usually accidental: a benchmark is published, it is quoted and discussed across the web, and the next training crawl picks it up. It can also be deliberate, when a developer trains on test data to climb a leaderboard. Either way, the benchmark stops measuring what it claims to.

Detection uses the training data when available and the model when not.

• N-gram overlap. Check whether sequences from each benchmark item appear verbatim in the training corpus. The GPT-3 paper flagged any test item sharing a 13-gram with training data; GPT-4's report used 50-character substrings. This is exact and cheap, but needs access to the training data, which an API user never has, and it misses paraphrases.
• Perplexity. A model assigns lower perplexity to text it has memorised. Compare perplexity on the benchmark to perplexity on fresh text of similar style and difficulty; an unusually low score is a contamination signal. This works without training data but is probabilistic.
• Behavioural probes. Ask the model to complete a benchmark item from its first half, or to reproduce the answer options in their original order. Verbatim reproduction is a giveaway.

GPT-3 on clean versus contaminated benchmark examples: contamination moved results by at most a few points on most benchmarks

The GPT-3 analysis above is both reassuring and sobering. For most benchmarks, scores on the "clean" subset were within a few points of the overall score, so contamination inflated results only modestly. But the clean subset was itself defined by an n-gram heuristic that misses paraphrases, and the study was run by the model's own creators.

Handling contamination. Reasonable practice is: assume every public benchmark is somewhat contaminated for every model trained after its release; weight recent or private benchmarks more heavily; prefer dynamic benchmarks that regenerate items or rotate test sets (LiveBench, LiveCodeBench) and human-preference leaderboards (Chatbot Arena) that cannot be memorised; and treat any public score that is much higher than performance on your private data as evidence of contamination rather than of capability. Your own evaluation set, kept out of any repository the model might crawl, is the only benchmark you can fully trust.

Designing your evaluation pipeline

The last stage of the funnel, and the thing you keep running after launch, is an evaluation pipeline built for your application. The book gives a three-step construction. Each step is simple to state and easy to skip, and skipping any of them produces a pipeline that reports numbers nobody should act on.

Step 1: evaluate every component and every turn

A production application is rarely one model call. A resume screener might be a PDF-to-text converter, an extraction prompt, a scoring prompt and a ranking rule. A customer-support assistant might be an intent classifier, a retriever, a reranker, a generator and a safety filter. End-to-end evaluation tells you the system failed; it cannot tell you which part failed, and the fix for a bad retriever is nothing like the fix for a bad generator.

So evaluate at three levels.

• Per component. Each component gets its own metric and its own test set: recall@k for the retriever, exact-match F1 for the extractor, entailment score for the generator against its context, block rate on a red-team set for the safety filter. Component evaluation is where most of the diagnostic power lives; fixing components bottom-up is usually faster than tuning the whole pipeline at once.
• Per turn. For conversational systems, evaluate each model response given the conversation so far. Is this turn relevant, faithful, safe and in character? Per-turn evaluation finds the point where a conversation goes wrong.
• Per task. Did the user get what they came for, however many turns it took? Was the support ticket resolved, the bug fixed, the booking made? Per-task evaluation is the number the business cares about, and it can only be measured on complete sessions.

Per-turn and per-task scores can disagree. An assistant that gives excellent individual answers but needs eight turns to resolve what should take two is failing the task. Measure both.

Step 2: create an evaluation guideline

An evaluation guideline is the document that defines what "good" means for this application concretely enough that two evaluators, human or model, would score the same output the same way. Without it, scores are noise. Teams consistently report it as the hardest part; LinkedIn's account of building their generative assistant named creating the guideline as the first hurdle.

A usable guideline has three parts.

Criteria specific to the application. Generic words ("helpful", "accurate") are not criteria until they are pinned down. For a support bot, "relevance" might mean "answers the question asked, using only information from the linked policy, without offering unrelated products". Decide what to do with correct-but-unwanted behaviour: a response that is factually right but in the wrong language, or that answers a question the business does not want answered, must have a defined score.

A scoring rubric with examples. For each criterion, define the scale and give at least one annotated example per level. A rubric without examples is a rubric every annotator interprets differently. Keep scales short: a 3- or 5-point scale with clear anchors beats a 10-point scale nobody can apply consistently, and binary pass/fail is often best for criteria that are verifiable.

Score	Faithfulness anchor
3	Every claim is entailed by the provided context
2	All claims are consistent with the context, but at least one is not supported by it
1	At least one claim contradicts the context, or a central claim is fabricated

A mapping from evaluation metrics to business metrics. The rubric scores mean nothing to a stakeholder until they are translated. If the assistant's faithfulness score is 80 percent, what fraction of support tickets can be automated without a human review? If it rises to 90 percent, how many more? Even a rough mapping ("at 95 percent faithfulness we can remove the human review step for tier-1 questions, saving X hours per week") turns an abstract score into a target and a budget justification. It also tells you when further improvement stops paying.

Step 3: define evaluation methods and data

With criteria and a rubric, decide how each criterion is measured and on what.

Methods. Use the cheapest method that is reliable enough for each criterion, and mix them. Verifiable constraints get code (regexes, JSON schema validators, unit tests). Classification-style criteria get a small specialised classifier or the model's logprobs where available. Open-ended criteria get an AI judge, calibrated against a human-labelled sample, with humans reserved for the hardest cases and for periodically re-checking the judge. Use comparative evaluation (pairwise preference) when the question is "is the new version better?" and pointwise scoring when the question is "is this good enough to ship?".

Keep three cheap signals in the pipeline even when a judge is available: automatic checks of format and length, a toxicity classifier, and logging of the raw outputs so that failures can be inspected rather than only counted.

Data. The evaluation set should look like production traffic, with deliberate over-sampling of the cases that matter most: hard queries, rare but costly categories, and known past failures. Start it from real logs if you have them, from expert-written examples if you do not, and from synthetic data to fill gaps; then annotate against the guideline, measuring inter-annotator agreement as you go. Low agreement is a guideline problem, not an annotator problem.

Slice-based evaluation

An aggregate score hides structure. Slice-based evaluation computes every metric separately for meaningful subsets of the data: query type, user segment, language, input length, document source, device, time of day, anything along which behaviour might differ. Slicing serves three purposes.

• Fairness and safety. A model that is 92 percent accurate overall and 70 percent accurate for non-native English speakers has a problem the aggregate conceals.
• Debugging. Slices localise failures. If faithfulness collapses only on documents over 20 pages, the fix is in chunking or context handling, not in the model.
• Avoiding Simpson's paradox. A model can win on every slice and lose in aggregate, or the reverse, if the models were tested on different mixes of easy and hard inputs.

Slice	Model A accuracy (n)	Model B accuracy (n)
Easy queries	95% (900)	98% (100)
Hard queries	60% (100)	65% (900)
Overall	91.5% (1,000)	68.3% (1,000)

Model B is better on both slices and worse overall because it was evaluated mostly on hard queries. Only the sliced view gives the right answer, and it also raises the real question: why were the two models tested on different distributions?

Define your slices from the guideline (which groups of users and inputs does the business care about?) and from failure analysis (where did the last version break?). Report every slice that is large enough to be statistically meaningful; flag the rest as under-sampled rather than silently merging them.

How much data is enough

Evaluation sets are expensive to annotate, so there is pressure to keep them small. Too small and the numbers do not mean anything: a 2-point improvement measured on 50 examples is one example changing sides. The rough statistics are unforgiving. For an accuracy-style metric, the standard error of a proportion is about sqrt(p(1-p)/n), so detecting a difference of roughly 10 percentage points needs on the order of a hundred examples, 3 points needs about a thousand, and 1 point needs about ten thousand.

The book's recommended tool is bootstrapping: resample your evaluation set with replacement many times, compute the metric on each resample, and look at the spread. If the metric varies by 5 points across bootstrap samples, your set cannot distinguish models 5 points apart; grow it, or accept that smaller differences are noise. Bootstrapping is free, uses your real data and distribution, and works for any metric, including judge-based ones.

When you compare two systems, use a paired comparison on the same inputs rather than two independent runs, and report the confidence interval of the difference. Paired comparison removes the variance from input difficulty and is far more sensitive for a given set size.

Interpreting results and iterating

A number without a baseline is not a result. Always report alongside a reference point: the previous version, a trivial baseline (always predict the majority class, return the first retrieved document), and where possible a human baseline. Report variance: judges are stochastic, so run them more than once and report the spread; models are sampled, so fix the temperature or report across seeds.

Watch for the evaluation pipeline being wrong rather than the system. Validate the judge against human labels on a held-out sample periodically, because judge accuracy drifts when the system's output distribution changes. Check that evaluation scores still correlate with the business metrics they were mapped to; when they diverge, the guideline needs revisiting before the model does.

Then iterate. Update the evaluation set when production failures reveal gaps, when features are added, when the user base shifts, and when the model is swapped. Version the evaluation set, the guideline and the judge prompts together with the application code, so that any score can be traced to the exact configuration that produced it. As the pipeline matures, add the production signals (user ratings, edits, escalations) that close the loop between offline evaluation and real outcomes.

Design trade-offs

Decision	Option	Choose it when	Cost you accept
Model source	Proprietary API	Fast time to market, frontier quality, small team, moderate volume	Data leaves your infrastructure, silent model changes, limited logprobs and finetuning
	Self-hosted open-weight	Strict privacy or residency, on-device, high stable volume, need for finetuning or internals	GPU infrastructure, inference engineering, on-call burden
Factuality check	Self-verification	No knowledge source available, short answers	N generations per check; consistently wrong answers pass
	Knowledge-augmented	Long-form, global facts, high stakes	Retrieval and judge cost; slow
	Entailment classifier	RAG and summarisation (local consistency)	Needs a good NLI model; context length limits
Instruction-following measurement	Verifiable instructions (IFEval style)	Format, length, keyword constraints	Covers only programmatically checkable constraints
	Judge-decomposed (INFOBench style)	Style, content and tone constraints	Judge cost and judge error
Benchmark use	Public leaderboard	Shortlisting from hundreds of models	Contamination, saturation, averaging that hides your criterion
	Private evaluation set	Any decision you will act on	Annotation cost; must be maintained
Comparison mode	Pairwise preference	"Is the new version better?"	Does not tell you whether either is good enough
	Pointwise rubric score	"Is this good enough to ship?"	Harder to calibrate across annotators
Eval set size	Small (hundreds)	Early iteration, large expected effects	Cannot detect differences under roughly 10 points
	Large (thousands+)	Launch gates, small expected effects, many slices	Annotation cost; slower to refresh

Interview angle

• "How would you decide between an API model and self-hosting an open-weight model for a customer-support assistant that sees customer PII?" Start with hard attributes: PII and any residency requirement decide whether prompts can leave your infrastructure; if they cannot, you self-host or use a provider that deploys into your VPC with zero retention. If an API is allowed, compare total cost (per-token at projected volume versus GPUs plus engineering), functionality you need (finetuning, logprobs, structured outputs) and tolerance for silent model updates. A common answer is API for launch with a contractual zero-retention tier, and a migration plan to open-weight once volume and quality bars are known.

• "What is the difference between local and global factual consistency, and how would you measure each?" Local consistency is agreement with supplied context; measure it with an entailment classifier or an AI judge given both texts. Global consistency is agreement with the world; measure it by decomposing the output into claims, retrieving evidence for each and judging support, as in SAFE or FActScore. Self-verification (sampling several answers and checking agreement) is a cheaper proxy for either when no source is available.

• "A new model tops the Open LLM Leaderboard. Would you switch to it?" Not on that basis. A leaderboard average weights benchmarks that may not matter to you, double-counts correlated ones, may reflect contamination, and says nothing about your prompts, latency or cost. Use the ranking to add the model to the shortlist, run it through your private evaluation set under your traffic profile, and compare with a paired test on the same inputs.

• "Why evaluate components separately when the user only sees the end result?" Because end-to-end evaluation cannot localise failures. If a RAG answer is wrong, the fix depends on whether the retriever missed the document, the reranker buried it, or the generator ignored it. Component metrics (recall@k, precision@k, faithfulness) tell you which, and components can be improved and regression-tested independently.

• "Your evaluation set has 80 examples and the new prompt scores 3 points higher. Do you ship?" Three points on 80 examples is about two examples changing outcome, well inside noise. Bootstrap the set to see the spread, or run a paired comparison with a confidence interval. Either grow the set to the hundreds or thousands needed for that effect size, or decide on other grounds.

• "What goes wrong if you only report the aggregate score?" You miss slices where the system fails (a language, a document type, a user segment), you cannot debug, and you are exposed to Simpson's paradox, where the aggregate ranking reverses the per-slice ranking because the models were tested on different mixes. Always report per-slice metrics for the slices the business cares about.

• "How do you detect that a benchmark has been contaminated?" With training data: n-gram overlap between benchmark items and the corpus (13-grams or 50-character substrings in the GPT-3 and GPT-4 reports). Without it: compare perplexity on benchmark items to fresh text of similar style, or probe whether the model can complete items verbatim. Treat a large gap between public scores and private-set performance as a red flag.

• "How would you design the evaluation pipeline for a RAG assistant from scratch?" Write the guideline first: criteria (faithfulness to retrieved context, relevance, safety, format), a 3-point rubric with examples per level, and a mapping to the business metric (tickets closed without escalation). Build an eval set from logs plus expert-written hard cases, annotated with agreement measured. Evaluate components (retriever recall@k, reranker precision@k, generator entailment score) and the task (resolution rate), with slices by query type and document source. Use code for verifiable checks, a calibrated AI judge for faithfulness and relevance, and a human sample to recalibrate the judge. Bootstrap to size the set, pair comparisons for changes, and schedule the whole thing to run against the live model to catch drift.

Pitfalls and misconceptions

• Treating "open source" as one category. Most open models are open-weight with licenses that restrict use; very few release training data. Read the license as a hard attribute and do not assume you can audit the data.
• Optimising cost before proving quality. Teams pick a small cheap model first, fail, and conclude the task is impossible. Establish that the task is solvable with a strong model, then descend.
• Trusting a leaderboard average. It equal-weights benchmarks you may not care about and dilutes uncorrelated ones like truthfulness. Build a custom leaderboard from the benchmarks that map to your criteria.
• Assuming the API model you evaluated is the one you are running. Hosted models are updated silently; schedule continuous evaluation and pin versions where possible.
• Evaluating only end to end. Without component metrics you cannot tell a retrieval failure from a generation failure, and you will tune the wrong thing.
• Writing a rubric without examples. Every annotator and every judge will interpret the scale differently; inter-annotator agreement will be low and the scores meaningless.
• Declaring a winner on a tiny eval set. A few points on tens of examples is noise. Bootstrap, pair the comparison, and report intervals.
• Forgetting that the evaluation pipeline is a system too. Judges drift, guidelines go stale, slices become under-sampled. Validate the judge against humans and the metrics against business outcomes on a schedule.
• Confusing instruction following with capability. A model can know the domain and still break your JSON schema. Measure both, separately.

Key terms

Term	Meaning
Hard attribute	A model property you cannot change (license, training-data disclosure, deployability, privacy policy); used as a filter
Soft attribute	A property you can improve with prompting, finetuning or system design (accuracy, faithfulness, latency, cost)
Open-weight model	A model whose weights are published but whose training data and full recipe are not; license may restrict use
Faithfulness / factual consistency	Whether an output is supported by its source (local) or by the world (global)
Hallucination	An output not grounded in the provided context or in fact
Self-verification	Checking factuality by sampling several responses and measuring agreement among them (SelfCheckGPT)
Knowledge-augmented verification	Decomposing an output into claims and verifying each against retrieved evidence (SAFE, FActScore)
Textual entailment (NLI)	Classifying a premise-hypothesis pair as entailment, contradiction or neutral; used to check local consistency
Instruction-following capability	How reliably a model honours format, length, content and style constraints; measured by IFEval, INFOBench
Verifiable instruction	A constraint whose satisfaction can be checked by code ("at least 300 words", "valid JSON")
TTFT / TPOT	Time to first token (prefill latency) and time per output token (decode latency)
Mean win rate	Leaderboard aggregation: the fraction of other models a model beats, averaged over benchmarks (HELM)
Data contamination	Benchmark test items present in a model's training data, inflating its score
N-gram overlap	Contamination check that looks for shared token sequences between benchmark and training data
Model drift	Change in a hosted model's behaviour over time without any change on your side
Evaluation guideline	The document defining criteria, rubric with examples, and mapping to business metrics
Per-turn vs per-task evaluation	Scoring each response in a conversation versus scoring whether the user's goal was achieved
Slice-based evaluation	Computing metrics separately on meaningful subsets of the data
Simpson's paradox	A trend that holds in every subgroup but reverses in aggregate because of unequal group sizes
Bootstrapping	Resampling the evaluation set with replacement to estimate the variance of a metric

Further reading

• AI Engineering, Chapter 4 ("Evaluate AI Systems"); the chapter summaries and resources in the book's repository: https://github.com/chiphuyen/aie-book
• Zhou et al., "Instruction-Following Evaluation for Large Language Models" (IFEval): https://arxiv.org/abs/2311.07911
• Manakul, Liusie and Gales, "SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models": https://arxiv.org/abs/2303.08896
• Wei et al., "Long-form factuality in large language models" (SAFE): https://arxiv.org/abs/2403.18802
• Min et al., "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation": https://arxiv.org/abs/2305.14251
• Lin, Hilton and Evans, "TruthfulQA: Measuring How Models Mimic Human Falsehoods": https://arxiv.org/abs/2109.07958
• Chen, Zaharia and Zou, "How is ChatGPT's behavior changing over time?": https://arxiv.org/abs/2307.09009
• Liang et al., "Holistic Evaluation of Language Models" (HELM): https://arxiv.org/abs/2211.09110
• Chiang et al., "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference": https://arxiv.org/abs/2403.04132
• Brown et al., "Language Models are Few-Shot Learners" (GPT-3; the contamination analysis is in the appendix): https://arxiv.org/abs/2005.14165
• Qin et al., "InFoBench: Evaluating Instruction Following Ability in Large Language Models" (2024)