Evaluation is the part of AI engineering that decides whether anything else worked. A classifier is scored with precision and recall against labels; a foundation model answers open-ended prompts where several different responses are all acceptable and the "right" one depends on the user. Chapter 3 lays out the methods available: the information-theoretic metrics used during training, exact and similarity-based scoring against references, using a model to judge another model, and ranking models by pairwise comparison.

The engineering stake is concrete. Without an evaluation pipeline you cannot tell whether a prompt change helped, whether a cheaper model is good enough, or whether last week's provider update broke your product. Teams routinely spend months on building and days on evaluation, then discover in production which one mattered. This chapter is about the methods; Chapter 4 applies them to whole systems.

In one minute

• Evaluation is hard because models are capable (hard to grade), outputs are open-ended (no single reference), tasks are unbounded (benchmarks saturate), and test data leaks into training (contamination).
• Entropy is the irreducible unpredictability of text; cross entropy is how many bits a model needs per token; perplexity is 2 to that power, the effective number of choices per token. Bits-per-character and bits-per-byte make models with different tokenizers comparable.
• Perplexity is a training and data-diagnostic signal, not a product metric: structured text and long contexts lower it, big vocabularies raise it, and post-training breaks its link to quality.
• Exact evaluation is deterministic: functional correctness (run the code, check the tests, report pass@k) when possible, otherwise similarity to references via exact match, lexical overlap (BLEU, ROUGE, METEOR, CIDEr) or semantic similarity on embeddings.
• Lexical metrics punish valid paraphrases; embedding similarity depends on the embedding model and says "similar", not "correct".
• AI as a judge scales human-like evaluation: fast, cheap, flexible, and in published studies agreeing with humans about as often as humans agree with each other. A judge prompt needs the task, explicit criteria, a scoring scheme and examples.
• Judges are inconsistent, criteria are ambiguous across tools, they add cost and latency, and they carry self-bias, position bias and verbosity bias. Specialized judges (reward models, reference-based judges, preference models) trade flexibility for cheaper, more consistent scores.
• Comparative evaluation asks "which is better" instead of "how good", then converts pairwise votes to a ranking with Elo or Bradley-Terry. Chatbot Arena is the canonical example. It scales poorly, assumes transitivity, lacks standardized criteria, and tells you relative, not absolute, quality.

Why evaluating foundation models is hard

Four properties make this harder than classical ML evaluation.

Capability. The better a model, the harder it is to tell a good answer from a great one. Checking a first-grade math worksheet is easy; grading a doctoral essay requires an expert and takes time. Many frontier outputs now exceed what a casual reviewer can verify, so evaluation itself needs expertise or tooling.

Open-ended outputs. "Explain quantum computing" has no single correct response. A response can be accurate but too technical, accessible but imprecise, well structured but incomplete. Classical metrics assume a label to compare against; here the label would have to be a rubric, and rubrics are themselves ambiguous.

Generality. The same model does code, summaries, dialogue and legal analysis. A task-specific benchmark covers a sliver of what it will be asked, and public benchmarks saturate as models catch up: GLUE was replaced by SuperGLUE within a year, and MMLU went from challenging to near-ceiling. Benchmark leadership says progressively less about your task.

Contamination. Public test sets end up in web crawls and therefore in pre-training data. A model can score well by having memorized the answers. Perplexity on the benchmark text is one detection tool (see below), but the safest defense is private evaluation data.

Compounding all of this, evaluation investment lags. Measured by papers and open-source repositories, evaluation tooling has grown rapidly since 2023, but still far more effort goes into models and applications than into knowing whether they work.

The number of papers on LLM evaluation published each month has grown rapidly

Cumulative number of evaluation repositories over time

The chapter's methods form a toolkit rather than a hierarchy. The flowchart below is the decision order used in practice; the rest of the chapter explains each branch.

Language modeling metrics

Language models are trained to predict tokens, and the metrics from that training (entropy, cross entropy, perplexity, bits-per-character, bits-per-byte) are the oldest evaluation tools. They do not measure usefulness, but they are indispensable for training diagnostics, data work and contamination checks, and interviewers like them because they test whether you understand what the model is optimizing.

Entropy

Entropy measures how much information, on average, each token in a language carries; equivalently, how unpredictable the next token is. If a language had two tokens that appeared with equal probability, each token would carry one bit of information and you would need one bit to encode it. English text, with a large vocabulary but strong regularities, has much higher entropy per word than that toy language and much lower entropy than a uniform draw over the vocabulary.

Formally, for a distribution P over tokens, H(P) = −sum_x P(x) · log2 P(x). Lower entropy means more predictable text (legal boilerplate, code), higher entropy means less predictable text (poetry, noisy chat). Entropy is a property of the data, not of any model: it is the floor no model can beat.

Cross entropy

A model Q trained on text from P is scored by cross entropy: the average number of bits it needs to encode tokens from P using its own predicted probabilities.

H(P, Q) = −sum_x P(x) · log2 Q(x) = H(P) + KL(P || Q)

The decomposition is the whole story. Cross entropy equals the data's entropy (irreducible) plus the KL divergence from the true distribution to the model's (reducible). A perfect model has KL zero and cross entropy equal to entropy; every mistake in the model's probabilities adds bits. Cross entropy is also the training loss: minimizing it on the training data is maximizing the likelihood of that data under the model. That is why a "loss curve" and an evaluation metric are often the same number.

Bits-per-character and bits-per-byte

Cross entropy is measured per token, and tokenizers differ, so two models' cross entropies are not directly comparable: a model whose tokenizer produces fewer, longer tokens has more work to do per token. Bits-per-character (BPC) normalizes by characters instead of tokens: multiply the per-token cross entropy by the number of tokens in the text and divide by the number of characters. Characters are also encoding-dependent (a Chinese character is one character but three UTF-8 bytes), so bits-per-byte (BPB) normalizes by bytes. BPB is the fairest number for comparing models across tokenizers and languages, and it has a clean interpretation: a model at 0.8 BPB can compress text to 80 percent of a bit per byte, which is why language modeling and compression are the same problem.

Perplexity

Perplexity is cross entropy exponentiated: PPL = 2^H(P, Q) when H is in bits, or e^H when H is in nats (the natural-log form most frameworks report). It reads as the effective number of equally likely choices the model faces per token. Perplexity 4 means the model is as uncertain as if it were picking uniformly among four tokens; perplexity 1 means it predicted every token with certainty.

Metric	Formula	Unit	Comparable across tokenizers?
Entropy	−sum P log2 P	bits per token	Property of data
Cross entropy	−sum P log2 Q	bits per token	No
Bits-per-character	cross entropy × tokens / characters	bits per character	Mostly
Bits-per-byte	cross entropy × tokens / bytes	bits per byte	Yes
Perplexity	2^H (bits) or e^H (nats)	effective choices per token	No

How to interpret these numbers

Perplexity moves for reasons that have nothing to do with model quality, so compare only like with like.

• Structured, predictable text has lower perplexity. HTML, code and legal text score lower than fiction. A low number on one dataset says nothing about another.
• Bigger vocabularies raise perplexity. More tokens to choose from means probability is spread thinner. A model with a 100K vocabulary and one with 32K are not comparable on raw perplexity; use BPB.
• Longer context lowers perplexity. More conditioning information makes the next token more predictable, so a model evaluated with a 4K window beats the same model at 512 tokens.
• Post-training raises perplexity. SFT and RLHF optimize for preferred responses rather than for predicting the training distribution, so chat models are miscalibrated as language models. Perplexity is a good signal for base models and a weak one for aligned models.
• Quantization raises perplexity. Reducing precision perturbs the probabilities; perplexity deltas are a standard way to measure how much a quantization scheme hurts.
• Lower perplexity is not better generation. A model can predict text well and still produce output that is unhelpful, repetitive or unsafe. Perplexity measures prediction, not production.

Where language modeling metrics are useful

1. Training diagnostics. Loss and held-out perplexity tell you whether pre-training or finetuning is converging, overfitting or diverging, long before any downstream eval runs.
2. Data quality and deduplication. Documents with very high perplexity under a reference model are often garbage; very low perplexity flags boilerplate and duplicates. Chapter 8 uses this for filtering.
3. Contamination detection. If a model's perplexity on a benchmark's test set is far below its perplexity on comparable fresh text, it has probably seen that test set. The same trick detects machine-generated text, which tends to be low-perplexity under the generating model.
4. Model comparison at fixed tokenizer. When comparing architectures, data mixes or training recipes with the tokenizer held constant, perplexity is the standard yardstick.

Exact evaluation

Exact evaluation produces a score with no subjective judgment in the loop: run the output, or compare it mechanically with a reference. It is deterministic and reproducible, and it should be used wherever the task permits.

Functional correctness

If the output does something, test whether it does the right thing. For code, this means executing the generated function against unit tests. Benchmarks such as HumanEval (164 Python problems with signatures, docstrings and hidden tests) and MBPP (about a thousand entry-level programming problems) work this way, and so should your own evaluation of a coding or text-to-SQL feature: run the SQL against a fixture database and compare result sets rather than comparing query strings.

The standard metric is pass@k: the probability that at least one of k sampled solutions passes all tests. pass@1 is the single-sample accuracy you would see in production; pass@10 or pass@100 shows what is reachable with best-of-N sampling and a verifier. The same principle extends beyond code: an agent that books a flight is evaluated by whether the booking exists, a game-playing bot by its score, a data extraction by whether the extracted fields match the source.

Functional correctness is unambiguous but not complete. Tests miss edge cases, a passing function can be slow or unreadable, and the test suite itself may be wrong. Generated code must also run in a sandbox: never execute model output with access to production systems, credentials or the network.

Similarity against reference data

When nothing executes, you compare the output with one or more reference responses (also called ground truth or canonical responses). This has a cost most teams underestimate: references must be written or verified by people, they can be wrong, and a task with many valid answers needs many references. Given references, there are four ways to compare.

Method	What it compares	Deterministic?	Typical use
Exact match	Whole strings after normalization	Yes	Short factual answers, labels, numbers
Lexical similarity	Surface overlap of words and n-grams	Yes	Translation, summarization, captioning
Semantic similarity	Distance between embeddings	Yes, given a fixed embedding model	Paraphrase-tolerant scoring of free text
Ask an evaluator	A human or AI judgment of equivalence	No	Anything the above handle badly

Exact match works when the answer is short and closed form: "1945", "Paris", "true". Normalize case, whitespace, punctuation and number formats before comparing, or "New York" and "new york" count as different. Variants accept any of several reference strings or check whether the reference appears inside the output.

Lexical similarity measures how much of the output's surface form overlaps with the reference. Fuzzy matching and edit distance work at the character level; n-gram overlap metrics work at the word level.

Metric	Measures	How	Designed for	Weakness
BLEU	n-gram precision	Fraction of candidate n-grams found in the reference, with a brevity penalty	Machine translation	Ignores recall and meaning; penalizes paraphrase
ROUGE	n-gram recall (ROUGE-N) or longest common subsequence (ROUGE-L)	Fraction of reference n-grams found in the candidate	Summarization	Rewards verbose outputs that happen to contain reference words
METEOR	Precision and recall with stemming and synonyms, plus a fragmentation penalty	Aligns words allowing morphological and synonym matches	Machine translation	More machinery, still surface-level
CIDEr	Consensus with several references	TF-IDF-weighted n-gram similarity across references	Image captioning	Needs multiple references; punishes correct but differently worded captions

The shared flaw is that meaning and wording are different things. "The cat sat on the mat" and "A feline rested upon the rug" share almost no n-grams. In the book's example, Fuyu's caption for a photo of Big Ben is a perfectly reasonable description yet scores only 0.4 CIDEr because it does not reuse the reference captions' words. Lexical metrics are fine for regression testing a translation system against fixed references; they are poor at judging an open-ended generator.

Fuyu's caption for a photo of Big Ben scores only 0.4 CIDEr against the reference captions, despite being a reasonable description

Semantic similarity compares meaning by turning both texts into embeddings and measuring how close the vectors are, usually with cosine similarity. Paraphrases score high, unrelated sentences score low, and no reference wording is privileged. BERTScore (token-level embedding matching) and MoverScore (optimal transport between embedding sets) are the best-known formulations. The method's limit is that it is only as good as the embedding model, and "semantically similar" is not "correct": a confident wrong answer about the same topic can sit close to the reference.

Introduction to embeddings

An embedding is a fixed-length vector that represents the meaning of an input (text, image, audio) such that similar inputs land near each other. Sizes are a few hundred to a few thousand dimensions: BERT base produces 768, OpenAI's text-embedding-3-small 1,536 and text-embedding-3-large 3,072 by default. The individual dimensions have no human-readable meaning; only the geometry (distances and directions) does.

Embeddings are produced by encoder models trained so that related pairs are pulled together and unrelated pairs pushed apart (contrastive learning), often on top of a masked language model like BERT. Models differ in training data, domain, language coverage, and whether queries and documents are embedded differently, so an embedding model is a component to evaluate, not a fixed utility.

Model family	Dimensions	Modalities	Notes
BERT / RoBERTa	768 or 1,024	Text	General encoders; sentence embeddings need pooling and further training
Sentence Transformers	384 to 1,024	Text	Contrastively trained for sentence similarity; strong open baseline
OpenAI text-embedding-3	1,536 or 3,072, truncatable	Text	Trained so shorter prefixes of the vector still work (Matryoshka)
Cohere Embed v3	1,024	Text, multilingual	Separate input types for queries and documents
CLIP	512 to 768	Text and image	Joint space enabling cross-modal retrieval and zero-shot classification
ImageBind, ULIP	varies	Image, text, audio, depth, 3D	Research joint spaces across many modalities

Joint embedding spaces

A joint embedding space maps different modalities into the same vector space, so a text query can be compared directly with an image. CLIP (OpenAI, 2021) trains an image encoder and a text encoder together on hundreds of millions of image-caption pairs with a contrastive objective: matching pairs get high cosine similarity, mismatched pairs low. That single idea enables zero-shot image classification (compare an image to the embeddings of "a photo of a dog", "a photo of a cat", and so on), text-to-image search, and automated evaluation of image generation by scoring how well an image matches its prompt (the CLIPScore idea). ULIP extends the approach to 3D point clouds and ImageBind to six modalities.

CLIP projects text and image embeddings into a joint space and trains on the cosine similarity of matching pairs

How to evaluate and introspect embeddings

You cannot read an embedding, but you can test it.

• Downstream performance. The standard answer: do retrieval, clustering, classification and semantic-similarity tasks get better? MTEB (Massive Text Embedding Benchmark) aggregates dozens of such tasks and is the leaderboard most teams consult. Your own retrieval set with labeled relevant documents is the one that matters.
• Nearest neighbors. Embed a sample of your data and inspect the top neighbors of random items. If "refund policy" sits next to "shipping times" rather than "return window", the model does not understand your domain.
• Separation of positive and negative pairs. Plot the cosine-similarity distribution for known-similar pairs against known-dissimilar pairs. Heavy overlap means similarity thresholds will not work, which matters for any "if similarity above 0.8 then match" logic.
• Projection. UMAP or t-SNE to two dimensions reveals whether your categories form clusters; it is a diagnostic picture, not a metric.
• Anisotropy and normalization. Many models produce vectors concentrated in a narrow cone so that every pair has similarity 0.7 or higher. Normalize vectors before cosine similarity, and calibrate thresholds on your data instead of assuming 0 means unrelated.
• Query-document asymmetry. Some models expect different prefixes or input types for queries and documents; using the wrong one silently degrades retrieval.

Embeddings return in Chapter 6 as the backbone of vector retrieval; the same checks apply there.

AI as a judge

Exact methods need an executable or a reference. Most product outputs have neither, which is why teams increasingly use a model to grade a model. AI as a judge (LLM-as-a-judge) treats a capable language model as a stand-in for a human evaluator. In LangChain's 2023 State of AI report, 58 percent of evaluations run on its platform used AI judges.

Why use a judge

• Speed and cost. Thousands of responses graded in minutes for a few dollars; humans at the same scale take weeks and real budgets.
• Flexibility. Any criterion you can describe in words becomes a metric: helpfulness, tone, adherence to a policy, factual consistency with a source, safety.
• No references required. A judge can score a response on its own, which exact methods cannot.
• Agreement with humans. In Zheng et al.'s MT-Bench study, GPT-4's agreement with human preferences was about 85 percent, above the 81 percent agreement between humans. Judges are not just acceptable substitutes; on some tasks they are more consistent than individual annotators.

An AI judge: the (question, answer) pair is slotted into a criteria prompt template and a model returns a score

ChatGPT scoring a paragraph 4.5 out of 5 and explaining its strengths and weaknesses

Three ways to use a judge

Mode	Inputs to the judge	Output	When
Score a response alone	Prompt, response, criteria	Score or label	No reference exists; open-ended quality, safety, tone
Compare to a reference	Prompt, response, reference answer	Correctness or match score	Factual QA with known answers but many valid phrasings
Compare two responses	Prompt, response A, response B	Which is better, or tie	Model selection, A/B of prompts, preference data

Pairwise comparison is usually the most reliable mode because relative judgments are easier than absolute ones for both people and models, and it sidesteps scale calibration (one judge's 4 is another's 3). Scoring alone is needed when you want per-response pass/fail decisions in production, such as a guardrail that blocks low-scoring answers.

How to prompt a judge

A judge prompt is a specification. The four components:

1. Task. What the judge is evaluating and the role it should take ("You are grading a customer-support reply for a bank").
2. Criteria. Define every term. "Relevant" should be spelled out as "addresses the user's actual question without adding unrelated information"; otherwise the judge supplies its own definition and you cannot compare scores over time.
3. Scoring scheme. Classification (pass/fail, A/B/tie), a small discrete scale (1 to 5) or a continuous score. Discrete scales with a description for each level work better than continuous ones; binary or three-way labels are the most consistent of all. Define what a 1 and a 5 look like.
4. Examples. One or two graded examples at different levels calibrate the judge and cut variance, exactly as few-shot examples do elsewhere.

Ask for the reasoning before the score. Reasoning first forces the judge to look at the evidence before committing; score first tends to produce post-hoc justification. Keep the judge's output structured (JSON with reasoning and score fields) so it can be parsed and aggregated, and use the lowest temperature available.

Different evaluation frameworks ship their own criteria, and the same name can mean different things. The table below paraphrases how four tools define overlapping criteria.

Criterion	Azure AI Studio	MLflow	LangChain	Ragas
Relevance	Does the answer address the question	How relevant is the output to the input	Is the output relevant to the input	How relevant is the answer to the question
Groundedness / faithfulness	Groundedness: is the answer supported by the provided context	Faithfulness: is the output grounded in the context	Not a default criterion	Faithfulness: are the claims supported by the retrieved context
Coherence	Is the response well organized and logical	Language quality and organization	Not a default criterion	Not a default criterion
Correctness	Not a default criterion	Not a default criterion	Correctness against a reference	Answer correctness against ground truth
Harmfulness	Content safety metrics	Not a default criterion	Harmfulness	Not a default criterion

The lesson is not which tool is right. Before adopting any framework's "relevance" score, read the prompt that produces it, pin the judge model version, and treat a change in either as a change in the metric.

Limitations of AI judges

Inconsistency. The same judge gives the same response different scores on different runs. Temperature 0 helps but does not eliminate it, and the judge's own biases (below) vary with phrasing. Treat judge scores as noisy measurements: use several samples for important decisions and report agreement with a human-labeled subset.

Criteria ambiguity. As the table shows, "relevance" is not one thing. Scores from different prompts, different judge models or different judge versions are not comparable, which undermines longitudinal tracking unless you freeze the judge. The author's rule: do not trust an AI-judge score unless you can see the judge's model and prompt.

Cost and latency. Every judged response is at least one extra model call, often with a stronger (more expensive) model than the one being evaluated. In production, judging every response as a guardrail doubles latency. Mitigations: judge a sample rather than everything, use a cheaper or finetuned judge for routine criteria, run judging asynchronously for monitoring, and reserve the strongest judge for offline evaluation.

Biases. Judges have systematic preferences that distort scores.

Bias	What happens	Mitigation
Self-bias	A model prefers responses from itself or its model family; GPT-4 judging GPT-4 outputs inflates them by roughly ten percentage points in reported studies	Judge with a different model family than the one evaluated, or average several judges
Position bias	In pairwise comparison, the first-listed response wins more often (humans, by contrast, tend to favor the most recent)	Run every pair in both orders; count a win only if it holds in both, otherwise tie
Verbosity bias	Longer, more detailed responses score higher regardless of accuracy	Instruct the judge to penalize padding, normalize for length, include concise high-scoring examples
Authority and format bias	Confident tone, headers and bullet points lift scores independent of content	Grade correctness and presentation as separate criteria

Position bias is the most reliably observed, and swapping order is the cheapest fix; verbosity bias matters most because the model being evaluated will learn to exploit it if judge scores feed training.

Search-augmented and decomposed judges

For factuality, a single holistic score is too coarse. SAFE (Search-Augmented Factuality Evaluator, Google DeepMind, 2024) decomposes a long response into individual facts, rewrites each to be self-contained, filters the ones irrelevant to the prompt, then checks each with web search before rating it supported or not. The same decompose-then-verify pattern (also used by FActScore) gives you a precision-style metric (fraction of claims supported) and a per-claim audit trail rather than one opaque number.

SAFE, a search-augmented factuality evaluator: the response is split into facts, each made self-contained, checked for relevance, then rated with web search

What models can act as judges

The judge does not have to be stronger than the model it grades. A reviewer can assess a paper they could not have written, and a small model can check format compliance, length, or whether a required disclaimer is present in a frontier model's output. Three patterns are common.

• Stronger model judges weaker. The default for offline evaluation: use the best model you can afford to grade candidates. Works well when the capability gap is large; agreement drops as the models converge.
• Self-evaluation. A model grades its own output. Cheap, useful as a sanity check for obvious failures (refusals, truncation, off-topic), but exposed to self-bias.
• Weaker or specialized model judges stronger. For narrow criteria, a cheap model or a finetuned classifier is more consistent and far cheaper than a general judge.

Specialized judges are small models trained for one evaluation job. They are cheaper and more consistent than prompting a general model, at the cost of flexibility.

Judge type	Input	Output	Trained on	Examples
Reward model	Prompt and one response	Scalar score	Human preference comparisons	RLHF reward models; Google's Cappy
Reference-based judge	Prompt, response and reference	Score or correctness label	Graded (response, reference) pairs	Prometheus, JudgeLM
Preference model	Prompt and two responses	Which is better, often with an explanation	Pairwise preference data	PandaLM

Reward models are the same objects used in RLHF (Chapter 2); evaluation reuses them as best-of-N selectors and quality monitors. Reference-based judges replace lexical metrics when references exist. Preference models directly implement pairwise comparison and feed the comparative methods below.

PandaLM, a specialized judge, compares two responses to the same instruction and explains which one is better

Ranking models with comparative evaluation

Pointwise versus comparative

Everything so far is pointwise: each response gets its own score. Comparative evaluation instead asks which of two responses is better and derives rankings from many such votes. People do this instinctively (ChatGPT periodically shows two drafts and asks which you prefer), and the method has a long history in sports and games.

ChatGPT asks users to compare two outputs side by side

Aspect	Pointwise	Comparative
Question asked	How good is this response, on a scale	Which of these two is better
Ease for the rater	Hard; needs a calibrated scale	Easy; relative judgment
Consistency across raters	Lower; scales drift	Higher
Scales with number of models	Linearly	Quadratically in pairs unless sampled
Tells you	Absolute quality against a threshold	Relative order
Saturation	Benchmarks saturate when everyone scores near the ceiling	Never saturates while new models arrive

From votes to a ranking

With n models there are n(n−1)/2 pairs, and each pair needs many votes across many prompts. The match log is converted into scores by a rating algorithm.

Elo. Borrowed from chess. Each model has a rating R. Before a match the expected score of A against B is E_A = 1 / (1 + 10^((R_B − R_A) / 400)); after it, R_A ← R_A + K · (S_A − E_A) where S_A is 1 for a win, 0.5 for a tie, 0 for a loss and K controls update size. Upsets move ratings more than expected results. Elo is online and simple, but it depends on match order, assumes a single skill dimension, and gives no uncertainty estimate.

Bradley-Terry. A statistical model in which each model i has a strength β_i and the probability that i beats j is P(i beats j) = e^(β_i) / (e^(β_i) + e^(β_j)). The strengths are fit by maximum likelihood (a logistic regression on the full match log), so all matches are used at once, order does not matter, and bootstrapping the log gives confidence intervals. Chatbot Arena switched from Elo to Bradley-Terry in late 2023 for exactly these reasons; the leaderboard's "Arena score" is a rescaled Bradley-Terry strength.

TrueSkill. Microsoft's Bayesian extension of Elo that tracks a mean skill and an uncertainty per player, so new entrants with few matches move fast and established ones settle. Useful when models join continuously; rarely used in published LLM leaderboards.

LMSYS Chatbot Arena

The best-known comparative evaluation is Chatbot Arena (LMSYS and UC Berkeley, launched May 2023, now LMArena). A visitor types any prompt, two anonymous models respond side by side, the visitor votes for one or declares a tie, and only then are the model identities revealed. Millions of votes have been aggregated with Bradley-Terry into a public leaderboard, with category breakdowns (coding, long queries, hard prompts) and per-language views.

Its strengths are real prompts from real users rather than a fixed test set, continuous updating as new models appear, and immunity to contamination (there is no test set to leak). Its weaknesses follow from the same design: the voter population skews toward enthusiasts, prompts are mostly short and conversational, votes reward what users like in a single turn (confidence, formatting, length) rather than correctness, and the result is a general-chat ranking that may not predict performance on your task.

Challenges of comparative evaluation

Scalability. Pairs grow quadratically and every pair needs enough votes to be statistically meaningful; a new model needs thousands of votes before its interval narrows. Arenas mitigate this by sampling pairs adaptively: prioritize matches between models with close or uncertain ratings, and skip pairs whose outcome is already certain.

Lack of transitivity. Elo and Bradley-Terry assume a single strength per model, so if A beats B and B beats C, A should beat C. Real preferences are multidimensional: a model better at code and worse at prose can lose to one model and beat another in a way no scalar explains. Rankings are therefore a projection, and the ordering of nearby models is often within noise.

Lack of standardization and quality control. Crowd voters receive no criteria. Each vote reflects that person's taste on that prompt, and voters cannot verify factual claims outside their expertise. Different arenas use different prompts, instructions and rating details, so scores are not comparable across them, and the same prompt distribution shift over time changes what the leaderboard measures.

From relative to absolute. A 60 percent win rate against the previous best model tells you nothing about whether either is good enough for your product. Comparative evaluation answers "which", not "how good". You still need pointwise evaluation against a usefulness threshold on your own data.

Where comparative evaluation is going

Comparative evaluation has one property no static benchmark has: it does not saturate while new models keep arriving. That makes it attractive as the durable public yardstick, and the trend is toward making it cheaper and more targeted. AI judges replace most human voters (Arena-Hard and AlpacaEval use GPT-4-class judges with length control to curb verbosity bias), domain-specific arenas appear for code, vision and search, and companies run private arenas on their own prompts to compare candidate models and prompt versions before shipping. The sound practice is to use public arenas to shortlist and your own pointwise evaluation to decide.

Design trade-offs: choosing an evaluation method

Situation	Method	Why
Output can be executed or checked mechanically (code, SQL, API calls)	Functional correctness, pass@k	Unambiguous, cheap, no judge needed
Short closed-form answers with references	Exact match with normalization	Deterministic; catches regressions precisely
Free text with references, fixed domain (translation, summarization regression tests)	Lexical metrics plus semantic similarity	Cheap trend signal; validate against humans once
Free text, paraphrase-tolerant scoring needed	Embedding similarity, BERTScore	Rewards meaning over wording; watch the embedding model
Open-ended quality, no references	AI judge with explicit rubric, reasoning first	Only scalable option; freeze model and prompt, calibrate on a human-labeled sample
Factual claims in long answers	Decomposed, search-augmented judging	Per-claim precision and an audit trail
Picking between models or prompt variants	Pairwise comparison with order swapping, Bradley-Terry	Most reliable judgment type; tells you which, not how good
Deciding whether to ship	Pointwise against a usefulness threshold on your traffic	Comparative ranking cannot answer this
Monitoring in production	Cheap specialized judge on a sample, async	Latency and cost constraints rule out heavy judges on every request
Comparing base models or data recipes	Perplexity or bits-per-byte at fixed tokenizer	The right tool for training-time diagnostics

In every row, the judge or metric is itself a model of quality and needs validation: pick a few hundred examples, have people label them, and measure how well your automated score agrees before you trust it at scale.

Interview angle

• Explain entropy, cross entropy and perplexity in one breath. Entropy is the inherent unpredictability of the data in bits per token. Cross entropy is the bits per token a model needs, equal to entropy plus the KL divergence between data and model. Perplexity is 2 to the cross entropy (or e to it in nats), the effective number of equally likely next tokens.
• Two models report perplexity 8 and 12. Which is better? Unknown without more information. Perplexity depends on tokenizer and vocabulary size, context length, dataset and whether the models are post-trained. Compare bits-per-byte on the same text with the same context, and only for base models.
• How would you detect benchmark contamination? Compute the model's perplexity on the benchmark's test items and on fresh text of the same style and difficulty. A large gap suggests memorization. Also check n-gram overlap between the benchmark and known training corpora, and keep a private test set as the final arbiter.
• Why is BLEU a bad metric for a chatbot? BLEU measures n-gram precision against reference wording and was built for translation, where references are tight. Chatbot answers have many valid phrasings; paraphrases score near zero and verbose text that reuses reference words scores high. It measures wording, not helpfulness or correctness.
• Design an evaluation for a RAG-based support assistant. Build a labeled set of real questions with reference answers and source documents. Use exact match on extractable facts, an AI judge with a rubric for groundedness (claims supported by retrieved context), relevance and tone, decomposed claim checking for factuality, and pairwise comparison with order swapping when choosing between prompt or model variants. Calibrate the judge on a human-labeled subset and track agreement over time; sample production traffic for ongoing judging.
• What goes wrong if you use the same model as both generator and judge? Self-bias inflates scores for its own outputs, and if judge scores feed training or selection the system learns the judge's preferences (verbosity, confident tone) instead of quality. Use a different model family or several judges, and spot-check against humans.
• A pairwise judge says A beats B 70 percent of the time. Is A ready to ship? No conclusion possible. Comparative results are relative. Check for position bias by swapping order, then evaluate A pointwise against the product's usefulness threshold on representative traffic.
• Compare Elo and Bradley-Terry for a model leaderboard. Elo updates sequentially after each match and depends on order; it is simple and online but gives no uncertainty. Bradley-Terry fits all matches jointly by maximum likelihood, is order-independent and supports bootstrap confidence intervals, which is why Chatbot Arena uses it. Both assume a single transitive strength per model.
• Why do arena rankings sometimes disagree with your internal evaluation? Arenas measure what enthusiast voters prefer on short general prompts in one turn, with no criteria and no fact-checking. Your task has its own distribution, correctness requirements and format constraints. Use arenas to shortlist, your evaluation to decide.
• How would you reduce the cost of AI-judge evaluation in production? Judge a random sample rather than every request, use a small finetuned judge or classifier for routine criteria, run judging asynchronously for monitoring, and reserve the strongest judge for offline evaluation and for escalations where a cheap judge is uncertain.

Pitfalls and misconceptions

• Treating perplexity as a quality score. It measures prediction on a given text under a given tokenizer and context; it is not comparable across those and says little about aligned chat models.
• Comparing perplexity across tokenizers. Use bits-per-byte. A larger vocabulary mechanically raises perplexity.
• Trusting benchmark scores without a contamination check. Public test sets leak into training data; a strong score may be memorization.
• Using lexical metrics to grade open-ended generation. BLEU and ROUGE penalize valid paraphrase and reward reference-word stuffing. Keep them for regression tests with tight references.
• Assuming embedding similarity equals correctness. A plausible wrong answer on the same topic can be close to the reference. Similarity is a feature, not a verdict.
• Adopting a framework's "relevance" metric without reading the prompt. Criteria names are not standardized; the judge prompt and model version define the metric. Freeze both.
• Scoring first, explaining second. Reasoning after the score becomes rationalization. Ask for reasoning first and parse a structured output.
• Running pairwise comparisons in one order. Position bias can decide the result. Swap and require agreement.
• Reading a leaderboard rank as "good enough". Comparative evaluation gives order, not absolute quality; only pointwise evaluation against your threshold answers the shipping question.
• Skipping human calibration. Every automated score, judge or metric needs a measured agreement with human labels on your data before you rely on it.

Key terms

Term	Meaning
Entropy	Average information per token in a data distribution; the unpredictability floor no model can beat
Cross entropy	Bits per token a model needs to encode data; equals entropy plus the KL divergence from data to model; also the training loss
KL divergence	The extra bits caused by using the model's distribution instead of the true one
Perplexity	Cross entropy exponentiated; the effective number of equally likely next tokens
Bits-per-character / bits-per-byte	Cross entropy normalized by characters or bytes instead of tokens, so models with different tokenizers can be compared
Contamination	Benchmark test data present in training data, inflating scores through memorization
Functional correctness	Evaluating by executing the output and checking the result, such as unit tests on generated code
pass@k	Probability that at least one of k sampled solutions passes all tests
Reference data	Human-written or verified canonical answers that outputs are compared against
Exact match	Whole-string equality after normalization
BLEU	n-gram precision against references with a brevity penalty; a translation metric
ROUGE	n-gram recall (ROUGE-N) or longest common subsequence (ROUGE-L) against references; a summarization metric
CIDEr	TF-IDF-weighted n-gram consensus across multiple references; a captioning metric
Embedding	A fixed-length vector representing an input's meaning; similar inputs are close in the space
Cosine similarity	Similarity of two vectors by the angle between them; the usual measure for embeddings
Joint embedding space	A shared vector space for several modalities, as in CLIP for text and images
MTEB	Massive Text Embedding Benchmark, the standard aggregate benchmark for embedding models
AI as a judge	Using a language model with a rubric prompt to score or compare responses
Self-bias, position bias, verbosity bias	Judges favoring their own family's outputs, the first-listed response, and longer responses
Reward model	A model trained on preference data to score a single response; used in RLHF and as a judge
Preference model	A model trained to pick the better of two responses, such as PandaLM
Comparative evaluation	Ranking by pairwise votes rather than independent scores
Elo	Sequential rating update from chess; order-dependent, no uncertainty estimate
Bradley-Terry	Probabilistic model of pairwise wins fit by maximum likelihood over all matches; used by Chatbot Arena
Transitivity	The assumption that if A beats B and B beats C then A beats C, required by single-score ranking models

Further reading

• AI Engineering, Chapter 3 (Chip Huyen, O'Reilly, 2025). Companion resources: https://github.com/chiphuyen/aie-book
• Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (2023), the source of the 85 percent agreement figure and the bias taxonomy: https://arxiv.org/abs/2306.05685
• Chiang et al., "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference" (2024): https://arxiv.org/abs/2403.04132
• Wang et al., "Large Language Models are not Fair Evaluators" (position bias, 2023): https://arxiv.org/abs/2305.17926
• Chen et al., "Evaluating Large Language Models Trained on Code" (HumanEval and pass@k, 2021): https://arxiv.org/abs/2107.03374
• Papineni et al., "BLEU: a Method for Automatic Evaluation of Machine Translation" (2002): https://aclanthology.org/P02-1040/
• Lin, "ROUGE: A Package for Automatic Evaluation of Summaries" (2004): https://aclanthology.org/W04-1013/
• Zhang et al., "BERTScore: Evaluating Text Generation with BERT" (2019): https://arxiv.org/abs/1904.09675
• Radford et al., "Learning Transferable Visual Models From Natural Language Supervision" (CLIP, 2021): https://arxiv.org/abs/2103.00020
• Muennighoff et al., "MTEB: Massive Text Embedding Benchmark" (2022): https://arxiv.org/abs/2210.07316
• Wei et al., "Long-form factuality in large language models" (SAFE, 2024): https://arxiv.org/abs/2403.18802
• Wang et al., "PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization" (2023): https://arxiv.org/abs/2306.05087
• Liu et al., "G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment" (2023): https://arxiv.org/abs/2303.16634
• The live Chatbot Arena leaderboard: https://lmarena.ai/