A foundation model on its own has two hard limits. It knows only what was in its training data, frozen at a cutoff, and it can only emit text. Chapter 6 covers the two patterns that remove those limits by building a system around the model. Retrieval-augmented generation (RAG) gives the model the right information at the right time by fetching it from an external store and placing it in the context. Agents give the model the ability to act: to call tools, observe results, plan multi-step work and correct course. Both are forms of context construction, which is why the chapter closes with memory, the mechanism that decides what stays in the context across a long interaction.

For an engineer these patterns are where most of the system-design work in an AI application lives. The model is a component you buy; the retriever, the tool layer, the planner loop and the memory store are components you design, evaluate and operate. Interview questions about "how would you build an assistant over our documents" or "how would you make the agent safe" are questions about this chapter.

In one minute

• RAG has two parts: a retriever that indexes documents and fetches the relevant pieces for a query, and a generator that answers using them. Retrieval quality bounds answer quality.
• Two retrieval families: term-based (TF-IDF, BM25; cheap, exact, lexical) and embedding-based (dense vectors, semantic, needs an ANN index such as HNSW, IVF, PQ, LSH or Annoy). Hybrid retrieval combines them, sequentially or with rank fusion.
• Compare retrievers with context precision and recall, and ranking-aware metrics (NDCG, MRR, MAP), alongside indexing cost and query latency.
• Retrieval optimisation levers: chunking (size, overlap, structure), reranking (cross-encoders, recency), query rewriting (resolve references, decompose) and contextual retrieval (prefix chunks with metadata or a model-written summary of their place in the document).
• RAG generalises beyond text: multimodal retrieval over images with joint embeddings, and text-to-SQL over tables.
• An agent is a model that perceives an environment and acts on it through tools. Tools either augment knowledge (retrievers, search, SQL), extend capability (calculators, code interpreters) or perform write actions (send, edit, pay), and write actions need human oversight.
• Planning should be decoupled from execution: a planner proposes, an evaluator validates, an executor runs. Function calling is the API form of plan steps; ReAct interleaves thought, action and observation; Reflexion adds explicit self-evaluation between attempts.
• Agents fail in three ways: planning failures (invalid tool, bad parameters, goal or constraint violations), tool failures (wrong output, bad translation to tool calls) and efficiency failures (too many steps, too much cost). Errors compound: at 95 percent per step, ten steps succeed about 60 percent of the time.
• Memory has three layers (internal weights, short-term context, long-term external store); manage it with FIFO, summarisation and retrieval, and watch for inconsistent memories.

RAG architecture

Retrieval-augmented generation fetches information relevant to a query from external sources and adds it to the model's context before generation. The name comes from Lewis et al. (2020), but the pattern is older: open-domain question answering systems of the 2010s already paired a document retriever with a document reader that extracted an answer span from the retrieved passages. The reader has become a generative model, and the pattern has become the default architecture for any application that must answer from data the model was not trained on.

Classic open-domain question answering: a document retriever fetches passages and a document reader extracts the answer

The retrieve-then-generate pattern: a retriever pulls relevant context from external memory and the generative model answers with it

Why retrieve instead of stuffing everything into the context, now that context windows reach a million tokens or more? Three reasons that do not go away as windows grow. Most corpora are far larger than any window (a company wiki, a codebase, a year of support tickets). Long prompts cost money and latency on every request. And models use long contexts unevenly: the lost-in-the-middle effect from Chapter 5 means that ten relevant paragraphs placed well beat a thousand paragraphs of which ten are relevant. Retrieval is a precision tool for constructing a context the model can actually use; long context makes the retrieval budget larger, not unnecessary.

The system has two components and two phases.

• The retriever has an indexing step, run offline whenever documents change, that processes documents into a searchable form, and a querying step, run online, that finds the most relevant pieces for a given query.
• The generator is the foundation model, prompted with the retrieved pieces and the query.

A basic RAG architecture: documents are split and embedded into a vector database at index time; at query time the retriever finds relevant chunks for the generative model

Retrieval is an old discipline (search engines and recommender systems are retrieval systems) and RAG inherits all of it. The difference is who consumes the results: a ranked list that is good for a human scanning ten links is not necessarily good for a model that will read the top five verbatim, so RAG tends to care more about precision at small k and about how the retrieved text is presented.

Retrieval algorithms

A retriever ranks documents by a relevance score between the query and each document. Algorithms differ in how they compute that score, and the two main families compute it from very different representations.

Term-based retrieval

Term-based (lexical, sparse) retrieval scores a document by the query terms it contains. The simplest version counts occurrences: term frequency (TF). That over-rewards common words, so TF-IDF multiplies by inverse document frequency, which down-weights terms that appear in many documents (the, and, system) and up-weights rare, informative ones (CUDA, deductible, Kubernetes). BM25, the standard today, refines TF-IDF with two corrections: a saturation term so that the twentieth occurrence of a word counts for less than the second, and document-length normalisation so that long documents are not favoured merely for containing more words. BM25 is the default scoring function in Lucene, Elasticsearch and OpenSearch.

Term-based systems also define what a "term" is. Tokenisation, stemming (running and runs match run), stop-word removal and n-grams (indexing "machine learning" as a unit, not two words) all shape relevance. Getting these right for a domain (product codes, chemical names, code identifiers) is where term-based tuning happens.

The strengths are real: exact matching on identifiers, error codes and rare names that embeddings blur; no model to run at index or query time; decades of mature infrastructure; and millisecond queries over millions of documents on commodity hardware. The weakness is equally real: no notion of meaning. "Terminate a process" does not match "kill a running program", and a query phrased differently from the document fails.

Embedding-based retrieval

Embedding-based (dense, semantic) retrieval converts both documents and queries into vectors with an embedding model, and ranks documents by vector similarity (cosine similarity or dot product). Documents are embedded at indexing time; the query is embedded at query time; the retriever returns the k nearest document vectors.

Embedding models are trained, usually with contrastive objectives, so that texts with similar meaning land close together regardless of wording. That is the point: "terminate a process" and "kill a running program" are neighbours. Quality depends heavily on the embedding model, which should be chosen (and ideally finetuned) for the domain and language; the MTEB benchmark is the usual starting point for comparison. A common refinement is a bi-encoder with separate query and document encoders, trained so that questions land near the passages that answer them (dense passage retrieval), which matters because a question and its answer rarely share surface form.

The costs are an embedding model to run at indexing time (over the whole corpus, and again whenever the model changes) and at query time, plus a vector database or index that can find nearest neighbours fast. The last part is the subject of the next section.

Vector search and ANN indexes

Finding the exact k nearest neighbours of a query vector requires comparing it to every stored vector: fine for ten thousand documents, impossible at a hundred million with a latency budget of tens of milliseconds. Approximate nearest neighbour (ANN) search trades a little recall for orders of magnitude of speed, and the main index families differ in what they trade.

Index family	Idea	Strengths	Costs and knobs
LSH (locality-sensitive hashing)	Hash vectors so that similar vectors collide in the same bucket; search only the query's bucket	Simple, fast inserts, sub-linear query	Lower recall for a given speed; many hash tables for good recall
HNSW (hierarchical navigable small world)	A layered proximity graph: sparse long-range links at the top, dense local links at the bottom; greedy descent from the top layer	Best recall-latency trade-off in practice; incremental inserts	High memory (graph plus full vectors); knobs M (links per node), efConstruction, efSearch
IVF (inverted file)	Cluster vectors with k-means; at query time scan only the nprobe nearest clusters	Good at scale, modest memory	Needs training on the data; recall drops if the right cluster is missed; nlist, nprobe
PQ (product quantization)	Split each vector into subvectors and replace each with the ID of its nearest centroid; store codes instead of floats	10x to 50x memory reduction; billion-scale	Lossy distances; usually paired with IVF (IVF-PQ) and a reranking pass over exact vectors
Annoy (trees)	A forest of random projection trees; each tree splits space with random hyperplanes	Memory-mappable, static indexes are tiny and shareable	No incremental updates; rebuild to add data

The engineering libraries that implement these (FAISS from Meta, ScaNN from Google, hnswlib, Annoy from Spotify) sit underneath most vector databases (Milvus, Weaviate, Qdrant, Pinecone, pgvector in PostgreSQL, Elasticsearch's dense vector fields). The database adds persistence, filtering by metadata, replication and an API; the index decides the recall-latency-memory triangle. For corpora under a few hundred thousand vectors, an in-process HNSW index is often simpler and faster than a separate database; for billions, IVF-PQ with a reranking stage is the usual shape.

Three knobs recur in every deployment: the recall target (what fraction of the true top-k the index returns; 0.95 is a common floor), the latency budget at the required queries per second, and memory (vectors are 1,536 or 3,072 floats each; a hundred million of them in float32 is over half a terabyte before any index overhead). Quantisation (PQ, or simply storing int8 vectors) is how the third constraint is usually met.

Comparing retrieval algorithms

Retrieval quality is measured on a labelled set of (query, relevant documents) pairs. Two families of metrics matter.

Set-based metrics look at the retrieved set of k documents without caring about order.

• Context precision (precision@k): the fraction of retrieved documents that are relevant. Low precision means noise in the prompt.
• Context recall (recall@k): the fraction of relevant documents that were retrieved. Low recall means the answer may not be in the context at all.

Rank-aware metrics reward putting the relevant documents near the top, which matters because the generator reads the first chunks most carefully and because you may truncate.

• MRR (mean reciprocal rank): the average of 1/rank of the first relevant result. Good when one right document suffices.
• NDCG (normalised discounted cumulative gain): sums graded relevance with a logarithmic discount by position, normalised by the ideal ordering. Handles multiple relevant documents of different quality.
• MAP (mean average precision): the mean over queries of precision averaged at each relevant position.

Beyond quality, compare indexing cost (embedding a corpus costs model time and money; BM25 indexing is nearly free), query latency and throughput, update behaviour (how quickly a new or edited document becomes searchable) and memory. The book's practical guidance is that term-based retrieval is a strong, cheap baseline that many teams never need to leave, that embedding-based retrieval wins on semantic queries at the price of infrastructure, and that the right comparison is on your queries, because the two approaches fail on different query types.

A retrieval evaluation set is also the most reusable artefact in a RAG project. Once you have one, every change to chunking, embeddings, index parameters or reranking can be measured in minutes without involving the generator.

Hybrid retrieval

Since the two families fail differently, hybrid retrieval combines them, in one of two shapes.

• Sequential. Use the cheap retriever first to narrow the corpus (BM25 returns a few hundred candidates), then the expensive one to rerank them (embedding similarity or a cross-encoder). Cost is bounded by the first stage.
• Parallel with fusion. Run both retrievers over the whole corpus and merge the ranked lists. Reciprocal rank fusion (RRF) is the standard merge: each document's score is the sum over lists of 1/(c + rank), with a constant c around 60, so a document ranked well by either retriever rises. Learned fusion weights are an option when you have enough labelled data.

Dimension	Term-based (BM25)	Embedding-based	Hybrid
Matches	Exact terms, identifiers, rare names	Meaning, paraphrase, cross-lingual	Both
Indexing cost	Negligible	Embedding every chunk	Both
Query latency	Milliseconds	Embed query plus ANN search	Highest, unless sequential
Infrastructure	Inverted index (Lucene family)	Embedding model plus vector index	Both, plus fusion logic
Typical failure	Vocabulary mismatch	Misses exact strings; embedding model blind spots	Fusion weights tuned for the wrong query mix

Hybrid is the sensible production default when queries mix natural-language questions with codes, names and quotes, which is most enterprise search. Start with BM25 plus one good embedding model fused by RRF, measure on your evaluation set, and only add complexity (learned fusion, a cross-encoder stage) when the numbers say the baseline is the bottleneck.

Retrieval optimization and RAG beyond text

The retrieval algorithm is one lever. Four others usually move the quality metrics more, and they are cheaper to try.

Chunking strategies

Documents are split into chunks because embeddings have input length limits and because the generator wants focused passages, not whole manuals. How you split is a first-order decision.

Strategy	Split rule	Pros	Cons
Fixed size	Every N characters or tokens, optionally with overlap	Trivial, predictable chunk count and size	Cuts sentences and tables mid-way; ignores structure
Sentence or paragraph	At natural boundaries	Chunks are coherent units	Variable size; very long paragraphs still overflow
Recursive	Try the largest separator first (sections, then paragraphs, then sentences), descend only when a piece is too big	Respects structure while bounding size; good default	More logic; still blind to meaning
Structure-aware	Follow headings, code blocks, table rows, HTML or Markdown structure	Keeps tables and code intact; carries headings as context	Needs a parser per format
Semantic	Split where consecutive sentence embeddings diverge	Topic-coherent chunks	Costs embeddings at index time; harder to tune

Three parameters interact.

• Chunk size. Smaller chunks give more precise matches and more diverse retrieved sets, but lose surrounding context and multiply the number of vectors (and the index cost). Larger chunks preserve context and dilute the signal. Common operating points are 256 to 1,024 tokens, bounded above by the embedding model's limit and by how many chunks you can afford to place in the prompt.
• Overlap. Repeating the last few sentences of a chunk at the start of the next avoids losing facts that straddle a boundary, at the cost of duplicate content in the prompt.
• Unit. Splitting by tokens of the embedding model keeps chunks within its limit; splitting by characters is simpler but can overflow for dense scripts.

Chunking must be evaluated, not reasoned about: the best size depends on the documents, the queries and the embedding model, and the retrieval evaluation set from the previous section is how you find it.

Reranking

The first-stage retriever is tuned for recall over a huge corpus and so is necessarily cheap. Reranking applies a more expensive, more accurate model to the small candidate set it returns, reordering (and typically truncating) it before the prompt is built.

The dominant reranker is a cross-encoder: a model that reads the query and a candidate document together and outputs a relevance score. Because it attends across both texts it is far more accurate than a bi-encoder's independent embeddings, and far too slow to run over a corpus, which is why it only sees the top 50 to 200 candidates. Hosted rerankers (Cohere Rerank and others) and open cross-encoders exist; late-interaction models like ColBERT sit between bi- and cross-encoders in cost and quality.

Reranking is also where non-semantic signals enter. Recency matters for news, prices, policies and anything versioned: a time-decay term that penalises older documents is often more valuable than a better embedding. Authority, document type, user permissions and popularity can all be folded into the reranking score.

Query rewriting

The user's words are often a poor query. In a conversation they refer to earlier turns ("when was his last order?"), they are terse, they bundle several questions, or they use a vocabulary the documents do not. Query rewriting uses a model call to transform the raw query before retrieval.

Query rewriting: a model resolves references like "his" using the conversation history before the query hits the retriever

Typical rewrites: resolve pronouns and references from the conversation history (the figure above turns "his" into the customer's name); expand with synonyms or domain terms; split a compound question into sub-queries retrieved separately; or generate a hypothetical answer and embed that (HyDE), on the theory that an answer looks more like the target passage than the question does. Rewriting adds a model call to the critical path and can hallucinate (a confidently wrong hypothetical answer retrieves confidently wrong documents), so measure it on the evaluation set like any other component.

Contextual retrieval

A chunk ripped out of a document loses the information that made it interpretable: which document, which section, which product version, what the pronouns refer to. Contextual retrieval puts some of that back before indexing.

The simple version augments each chunk with metadata: the document title, section headings, tags, dates, and sometimes a list of questions the chunk answers, so that a query matching the question text retrieves the chunk. The stronger version, published by Anthropic in 2024, prepends to each chunk a short model-generated context sentence describing where the chunk sits and what it is about ("this chunk is from the Q2 report for ACME; the previous section discussed revenue"), then embeds and BM25-indexes the augmented chunk. In their experiments this cut retrieval failures by about half on its own and by about two thirds combined with reranking. It costs a model call per chunk at indexing time, which prompt caching makes affordable.

Contextual retrieval: each chunk is prefixed with a model-written summary of where it sits in the document before being embedded and indexed

RAG beyond text

The retrieve-then-generate pattern does not depend on the data being prose.

Multimodal RAG. If the knowledge is in images, diagrams or video frames, a joint text-image embedding model (CLIP and its descendants) places images and text in one vector space, so a text query can retrieve images and a multimodal generator can answer from them. The retriever draws from a text database and an image database and the generator receives both kinds of context. Captioning images into text at index time and retrieving the captions is the cheaper, lossier alternative.

Multimodal RAG: the retriever draws on both a text database and an image database to answer a question

Tabular data and text-to-SQL. Much enterprise knowledge is in tables, and the right way to "retrieve" from a table is to query it. A text-to-SQL step turns the question into a SQL query against the relevant table, the query is executed, and the result rows are handed to the generator. Two sub-problems appear at scale: table selection when there are hundreds of tables (retrieve candidate schemas first, by name and column descriptions, then write SQL against the shortlist), and query safety (read-only credentials, query timeouts and row limits, because a generated query is untrusted code).

RAG over tabular data: a text-to-SQL step queries the table and the result is handed to the generative model

Text-to-SQL is a bridge to the agent half of the chapter: the model is no longer only reading context but deciding on an action (which query to run) whose result becomes its context. Retrieval, viewed this way, is just one tool.

Agents: environment, actions and tools

What an agent is

An agent, in the classical AI sense, is anything that perceives its environment and acts upon it. For an AI agent, the environment is defined by the use case (a codebase and terminal for a coding agent, the web for a research agent, a game for a game-playing agent, a set of business APIs for a workflow agent) and the actions are the tools the agent is allowed to call. The model is the brain; the environment is the world it acts in; the tools define what it can do there.

SWE-agent (2024) is a clean example. The environment is a computer with a repository and a shell. Instead of giving the model a raw terminal, its designers built an agent-computer interface (ACI): a small set of commands for navigating, searching, viewing a file window and editing with line numbers, each returning concise, model-friendly feedback (including linter errors after an edit). The interface, not the model, was the main source of their gains, which is a lesson that generalises: design tools for the model that will use them, not for a human who already knows the shell.

SWE-agent: an agent-computer interface gives the model LM-friendly commands and feedback for navigating, searching and editing a repository

Compounding errors and write actions

Agents need stronger models than single-call applications, for two reasons. Errors compound: an agent that is right 95 percent of the time on each step completes a ten-step task about 60 percent of the time (0.95 to the tenth power), and 70 percent per step gives under 3 percent. And stakes rise: an agent with tools can cause harm that a chatbot cannot.

That second point gives the most important design distinction in the chapter. Read actions observe the environment (search, fetch, query, list) and are safe to retry and to automate. Write actions change it (send, edit, delete, pay, deploy) and are often irreversible. Huyen's analogy is that you would not let an intern delete the production database, so you should not let an unreliable model initiate a bank transfer. Give write capabilities incrementally, gate the consequential ones behind human approval, and remember from Chapter 5 that any content the agent reads may be trying to make it write.

Tool roles

An agent's tool inventory determines what it can know and do. The book sorts tools into three roles.

Knowledge augmentation. Tools that bring information into the context: text and image retrievers over private data, SQL executors, web search and browsing, internal APIs (inventory, CRM, calendar), message and email readers. Web access is the most powerful and the riskiest knowledge tool, because it exposes the agent to anything anyone has published, including content written to hijack it.

Capability extension. Tools that do what the model does badly: calculators and unit converters, calendars and time-zone converters, translators, and above all code interpreters, which let the model write and run code to analyse data, make charts or test hypotheses. A code interpreter is a universal tool and also a universal attack surface, so it runs in a sandbox with no network and no access to secrets.

Write actions. Tools that change the world: sending email, updating records, executing transactions, committing code, controlling devices. These are what make agents useful for automation, and they are where the safety design concentrates.

Tool role	Examples	Primary risk	Standard control
Knowledge augmentation	Retriever, SQL (read), web search, API lookups	Injected or wrong content enters the context	Treat results as untrusted data; read-only credentials
Capability extension	Calculator, code interpreter, translator	Code execution, resource abuse	Sandbox, timeouts, no network or secrets
Write actions	Email, DB writes, payments, deployments	Irreversible harm, acting on injected instructions	Least privilege, human approval, audit log, reversibility where possible

More tools are not better. Each added tool makes tool selection harder for the model, lengthens the prompt and widens the attack surface. Toolformer (2023) showed a model could learn to call a handful of APIs; Chameleon (2023) composed about a dozen tools; in practice, inventories beyond that need hierarchical selection (pick a category, then a tool) or retrieval over tool descriptions. Start small, add tools when the evaluation shows a gap they would fill, and check what each tool is actually used for before keeping it.

Planning and agent evaluation

Given a goal and an inventory, the agent must decide which tools to call, in what order, with what arguments. That is planning, and it is the hardest and least reliable part of an agent.

Planning overview

A plan is a roadmap from the current state to the goal: a sequence (or graph) of actions. The book's first and most practical recommendation is to decouple planning from execution. If the agent executes each step as it thinks of it, a bad plan burns real tool calls, time and possibly irreversible actions before anyone notices. Instead:

Decoupling planning from execution: a planner proposes a plan, an evaluator checks it, and only then does the executor call tools

• The planner produces a plan.
• The evaluator validates it before anything runs. Validation can be heuristic (every tool exists, every parameter is well-typed, step count under a limit, no forbidden action) or a model judge scoring plausibility, or a human for high-stakes tasks. An invalid plan goes back to the planner with the reason.
• The executor runs the validated plan, step by step, feeding observations back.

Two more components usually sit alongside. An intent classifier in front of the planner routes out-of-scope requests away before any planning happens (an e-commerce agent should refuse to plan a tax return). And a human-in-the-loop review point, either on the plan or on specific write steps, is how most production agents stay safe. The planner, evaluator and executor can be the same model with different prompts, different models, or separate agents in a multi-agent system; the architecture is the same.

Foundation models as planners

Whether language models can plan at all is contested. Subbarao Kambhampati and colleagues argue that planning is search over a state space with backtracking, that an autoregressive model does neither, and that what looks like planning is approximate retrieval of plans resembling those in the training data: plausible to a lay reader and wrong at execution time. Their constructive proposal, LLM-Modulo, keeps the model as a generator of candidate plans and adds external verifiers (simulators, constraint checkers, formal planners) in a loop, which is the planner-evaluator-executor pattern with a strict evaluator.

The engineering stance the book takes is pragmatic. Treat model-generated plans as proposals, validate them with everything you can check deterministically, give the model the information it needs to plan (the tools, their constraints, the current date, the user's permissions) and let it backtrack by feeding failures back. This is also why reinforcement learning planners are a contrast rather than a competitor: an RL planner is trained for one environment through trial and error and is excellent there, while a foundation-model planner is prompted and generalises across environments at lower reliability. The lines are blurring as foundation models are finetuned with RL on agentic tasks.

Plan generation and function calling

In practice, plans are produced by prompting the model with the tool inventory and asking for a sequence of calls. Most APIs standardise this as function calling: you pass a list of tools, each with a name, a description and a parameter schema; the model returns the name and arguments of the call it wants to make; your code executes it and returns the result; the loop continues until the model answers in plain text.

The model never executes anything; it emits a structured request and your application decides whether to honour it. That boundary is where validation and permission checks live. Most APIs also let you control the mode: force a specific tool, forbid tools, or let the model choose.

Function calling: define the tools, describe them to the model, then let the model pick which tool to invoke for a query

Three things make function calling reliable. Write tool descriptions for the model, with explicit guidance on when to use (and not use) each tool. Give the model the context it needs to fill parameters correctly: the current date for "last week", the user's ID, units and formats. And inspect parameter values: the single most common agent bug is a valid tool called with a plausible but wrong argument (the wrong date range, the wrong table, an empty query). Log every call with its arguments, and in evaluation score argument correctness separately from tool choice.

Planning granularity. A plan can be expressed as exact function calls (get_time(), fetch_top_products(start, end)) or as natural-language steps ("get the current date; find the top-selling products for the last week"). Fine-grained plans are directly executable but hard to generate correctly and brittle when a tool's API changes. Coarse plans are easier to generate and robust to API changes, but need a translator step to turn each line into a concrete call, and the translator can be wrong. Hierarchical planning uses both: a high-level plan in natural language, then a low-level planner that expands each step into calls, with validation at each level.

Granularity	Example	Pros	Cons
Exact function calls	search_db(query="Q3 revenue", table="financials")	Executable as-is, precisely checkable	Hard to generate; breaks when tool signatures change
Natural-language steps	"Look up Q3 revenue in the financial database"	Easier to generate; survives API changes; readable by humans	Needs a translator; translation errors
Hierarchical	High-level steps, each expanded into calls	Combines robustness with precision	Two planners to evaluate

Complex plans and control flows. Real tasks are not straight lines. Plans need sequential steps (each uses the previous result), parallel steps (independent lookups run at once, then merge), conditionals (if the search returns nothing, broaden it; if the order is over a threshold, require approval) and loops (for each document, extract the dates). Agent frameworks differ in which control flows they support natively, and the more expressive the plan language the harder the plan is to generate, validate and evaluate. Match the plan structure to the task; a linear plan that works beats a general one that sometimes does.

Control flows an agent may need: sequential, parallel, if-branches and loops, with an example of each

Reflection and error correction

Plans fail on contact with the environment. Reflection is the agent examining its own progress (is this step done? did that result make sense? is the goal reached?) and adjusting, and it is what separates an agent that recovers from one that marches through a broken plan.

ReAct (Yao et al., 2022) builds reflection into every step. The agent alternates a thought (reasoning about the situation and the next action), an action (a tool call) and an observation (the tool's result), repeating until it decides it is done. Making the reasoning explicit before each action improves action choice and makes the trajectory auditable: you can read why the agent did what it did.

An agent following the ReAct framework interleaves thoughts, actions and observations on a multi-hop question

Reflexion (Shinn et al., 2023) separates two roles that ReAct blends. An evaluator judges the outcome of a whole attempt (did the tests pass? was the answer correct?), and a self-reflection module writes a verbal diagnosis of what went wrong and what to try differently. The reflection is stored in memory and included in the next attempt's prompt, so the agent learns across trials within a task without any weight update.

Reflexion separates the evaluator from the self-reflection step: the agent reflects on a failed trajectory before trying again

Reflection is also where the simplest guardrails live: checking a result against the question before answering, validating a planned action against constraints before executing, and comparing multiple sampled answers for consistency. All of it costs tokens and latency; an agent that reflects after every step is slower and dearer than one that reflects at checkpoints. Place reflection where errors are expensive and where there is information to reflect on.

Tool selection

Which tools to give an agent is an empirical question. The book's methods are the methods of any ablation study.

• Ablate. Remove a tool and measure the drop in task success; a tool whose removal changes nothing is a cost without benefit.
• Look at usage. Count how often each tool is called and in what context. Rarely used tools are candidates for removal; frequently misused tools need better descriptions.
• Look at transitions. A tool transition graph shows how often each tool follows each other. A pair that almost always appears together may be worth merging into one tool; a pair that never co-occurs may mean one is redundant.

A tool transition graph: how often each tool follows another, which reveals combinations worth merging or dropping

Different models prefer different tools. The Chameleon experiments showed that, on the same benchmarks, GPT-4 as the planner leaned on knowledge retrieval while ChatGPT leaned on image captioning and other tools; the inventory that is optimal for one planner is not optimal for another, so re-run the ablation when you change models.

Which tools Chameleon calls on two benchmarks, with ChatGPT versus GPT-4 as the planner: different models express different tool-use patterns

A more ambitious direction lets agents build their own tools: Voyager's skill library stores successfully executed code as reusable skills that later plans can call. It is powerful and it compounds the risk of untrusted code, so it belongs behind the same sandbox and review as any write action.

Agent failure modes and evaluation

Evaluating an agent is detecting its failures, and the failures fall into three groups.

Planning failures.

• Invalid tool: the plan calls a tool that does not exist in the inventory (often one the model saw in training).
• Valid tool, invalid parameters: wrong number or type of arguments.
• Valid tool, wrong parameter values: the hard one, because the call executes; the date range is off by a month, the query is empty, the wrong customer ID is used.
• Goal failure: all steps run and the goal is not met, or a constraint is violated (over budget, past deadline, used a forbidden action).
• Reflection failure: the agent declares success when it has not succeeded, or keeps looping on a task that is done.

Tool failures. The plan is right and a tool returns the wrong thing: a captioner mislabels an image, a text-to-SQL step writes a query that runs but answers a different question, an API returns stale data, a translator turns a correct high-level step into the wrong low-level call. Tool failures are diagnosed by evaluating each tool in isolation, exactly as RAG components were evaluated in Chapter 4.

Efficiency failures. The agent gets there, slowly: too many steps, too many tokens, too many dollars, too much wall-clock time. Compare against a baseline (a scripted solution, a human, a simpler agent) on steps per task, cost per task and time per task, and remember that a step count that is efficient for a model may be wrong for a human workflow, or the reverse.

Failure	Signature in logs	Mitigation
Invalid tool	Unknown function name	Validate against inventory before execution; better tool descriptions
Invalid parameters	Schema validation error	Strict JSON schema; retry with the error message
Wrong parameter values	Call succeeds, result off-target	Give the model the context it needs; echo parameters for a check; evaluate value accuracy
Goal or constraint failure	Task marked complete, checker fails	Explicit goal check; constraint validation in the evaluator
Reflection failure	Premature "done" or endless loop	Independent evaluator; step budget; loop detection
Tool failure	Correct call, wrong output	Per-tool evaluation sets; fallbacks; retries with backoff
Efficiency failure	High steps, tokens, latency	Step and cost budgets; plan validation; parallelise independent steps

To evaluate planning, build a dataset of (task, tool inventory) pairs with known good plans, run the planner, and measure: the fraction of generated plans that are valid, the fraction of tool calls that are valid, parameter-value accuracy, how many attempts it takes to reach a valid plan, and which invalid tools appear most often. Those numbers tell you whether to fix the prompt, the tool descriptions, the inventory or the model. Always run agents with a maximum step budget; a confused agent without one will loop until the bill stops it.

Memory

An agent that forgets everything between turns, or between sessions, is a bad agent; a coaching assistant to which you must re-explain your situation every time is close to useless. Memory is the mechanism for retaining and retrieving information, and the book describes it as three layers.

Layer	Where	Capacity	Persistence	Updated by
Internal knowledge	Model weights	Vast, fixed	Permanent until retraining	Training and finetuning
Short-term memory	Context window	Thousands to a couple of million tokens	This session only	Appending to the conversation
Long-term memory	External store (database, vector index, files)	Unbounded	Across sessions	Explicit reads and writes, often via retrieval

Three memory mechanisms: the context as short-term memory, external memory with a retriever as long-term memory, and training data as internal knowledge

Seen this way, RAG is long-term memory with a retriever, and the entire chapter is about what goes into the short-term memory on each call. A memory system does four jobs: it keeps a long interaction working after the context fills (overflow management), it persists information across sessions, it keeps the agent consistent with what it said and learned before, and it lets you enforce structure and integrity on what is stored (what is a fact, a preference, a task, a reflection).

Memory management

Three questions define a memory system: what to store, how to retrieve it, and what to drop or compress.

FIFO. The simplest overflow strategy keeps the most recent N messages and drops the oldest. Cheap, and wrong whenever the important information came early (the task statement, the user's constraints, their name).

Summarisation. Periodically compress the older part of the conversation into a summary that replaces it, keeping recent turns verbatim. This preserves the gist at the cost of detail (exact numbers, quotes, nuance), and the summary itself can drift as it is re-summarised.

Retrieval. Store everything (or everything judged important) in an external store and retrieve what is relevant to the current turn, exactly as RAG does for documents. This is the only strategy that scales to months of interaction, and it inherits every retrieval problem above.

Reflection-based storage. Rather than storing raw messages, have the model extract what matters (facts, preferences, goals, outcomes) into structured entries, with timestamps, sources and importance scores. The Generative Agents work (Park et al., 2023) combined a stream of observations with periodic reflections and retrieved by recency, relevance and importance; most production memory systems are a simplification of that design.

Consistency and structure

Stored memories conflict. A user who said they were vegetarian in March orders a steak in June; a preference recorded for one project is wrong for the next. Retrieval returns both and the model picks one, or worse, blends them. Memory systems need a policy for conflicts: prefer the most recent, ask the model to reconcile when writing, or surface the contradiction to the user. Timestamps and provenance on every entry are the minimum that makes any of these possible.

Structure helps. Unstructured text memories are flexible and easy to write but hard to query, de-duplicate and validate. Structured memories (typed fields, a schema for preferences versus facts versus tasks) are more work to extract and far easier to keep consistent, to expire and to audit, which matters when the memory contains personal data. Treat the memory store as a service with a clear read and write API, metadata on every record (time, source, confidence, expiry) and an eviction policy, and both agents and the compliance team will be able to reason about it.

Design trade-offs

Decision	Option A	Option B	Choose A when	Choose B when
Retrieval family	Term-based (BM25)	Embedding-based	Queries contain identifiers, codes, exact phrases; small team; no GPU	Queries are paraphrased natural language; cross-lingual; vocabulary mismatch dominates failures
Hybrid shape	Sequential (cheap first, rerank)	Parallel with RRF	Latency and cost bounded; large corpus	Both retrievers have independent strengths; recall matters most
ANN index	HNSW	IVF-PQ	Up to tens of millions of vectors; memory available; best recall-latency	Hundreds of millions or more; memory constrained; can afford a reranking pass
Chunk size	Small (256 tokens)	Large (1,024 tokens)	Precise fact lookup; many short questions	Questions need surrounding context; documents are narrative
Reranker	None	Cross-encoder	Retrieval precision already high; latency tight	Top-k is noisy; generator quality tracks retrieval precision
Long context vs RAG	Put the whole corpus in the prompt	Retrieve	Corpus is a few documents and static; per-request cost acceptable	Corpus is large, changing or permissioned; latency and cost matter
Plan granularity	Exact function calls	Natural-language steps plus translator	Small, stable tool set; reliability of generation is high	Tools change; plans must be human-readable; hierarchical planning
Reflection frequency	Every step (ReAct)	At checkpoints or per attempt (Reflexion)	Steps are cheap and each can mislead the next	Steps are costly; outcome can be judged at the end
Write actions	Autonomous	Human approval	Reversible, low stakes, well-tested	Irreversible, financial, external-facing, or reading untrusted content
Memory strategy	Summarisation	Retrieval over a store	Single long session; gist suffices	Many sessions; details and consistency matter

Interview angle

• "Design a question-answering assistant over our internal documentation." Index: parse documents structure-aware, chunk at 256 to 512 tokens with overlap, prefix chunks with title and section (contextual retrieval), store in both a BM25 index and a vector index. Query: rewrite the query with conversation history, run hybrid retrieval with RRF, rerank the top 100 with a cross-encoder, place the top five in the prompt at the start and end, instruct the model to answer only from context and cite chunk IDs. Evaluate the retriever on a labelled set with recall@k and NDCG, the generator on faithfulness, and the whole on task success, as in Chapter 4. Add permissions filtering at retrieval time, not in the prompt.

• "Compare BM25 and embedding retrieval. When does each fail?" BM25 scores lexical overlap with IDF weighting, saturation and length normalisation; it is cheap, exact and fails on vocabulary mismatch. Embedding retrieval compares dense vectors from a trained encoder; it captures meaning and fails on exact strings, rare identifiers and anything the embedding model was not trained on. Hybrid with rank fusion covers both failure modes.

• "Explain HNSW and when you would pick IVF-PQ instead." HNSW is a layered proximity graph; search descends from sparse top layers to dense bottom layers greedily, giving the best recall at low latency, with memory proportional to vectors plus graph links. At hundreds of millions of vectors memory becomes the constraint, so IVF clusters the space and PQ compresses vectors into short codes; you scan a few clusters, score approximately, then rerank candidates with exact vectors.

• "What is the difference between context recall and NDCG, and which matters more for RAG?" Recall@k asks whether the relevant documents are in the retrieved set at all; NDCG asks whether they are near the top, with graded relevance. For RAG both matter: recall because a missing document cannot be answered from, ranking because the generator reads the first chunks most carefully and you truncate. Report both; fix recall first.

• "How does function calling work and where do you put safety checks?" The application sends tool schemas with the messages; the model returns a structured call (name, arguments); the application validates and executes it and appends the result; the loop continues. Safety checks go at the boundary: schema validation, permission checks against the user, parameter sanity checks, and human approval for write actions, before anything executes.

• "Why decouple planning from execution?" Executing while planning wastes tool calls on bad plans and risks irreversible actions before validation. A planner-evaluator-executor loop lets heuristics, a judge model or a human reject invalid or unsafe plans cheaply, and feeds the rejection reason back to the planner.

• "What goes wrong with an agent that is 90 percent accurate per step?" Errors compound multiplicatively: a ten-step task completes about 35 percent of the time. Mitigations are fewer steps (better tools, hierarchical plans), validation between steps, reflection at checkpoints, and stronger models for the planning step.

• "Compare ReAct and Reflexion." ReAct interleaves thought, action and observation at every step within one attempt. Reflexion adds an evaluator that judges a whole attempt and a self-reflection step that writes a verbal lesson stored in memory for the next attempt. ReAct is per-step course correction; Reflexion is cross-attempt learning without weight updates.

• "How would you give a long-running assistant memory, and what breaks?" Three layers: the context for the current session, an external store for persistent facts, the model's weights for general knowledge. Write structured entries (type, timestamp, source, confidence) extracted by reflection; retrieve by relevance, recency and importance; summarise old turns in-session. What breaks: conflicting memories (define a resolution policy), stale preferences (expiry), privacy (treat the store as personal data with access controls and deletion).

• "An agent read a web page that told it to email the user's contacts. What should have stopped it?" The instruction hierarchy in the model (tool content is lowest privilege), delimiting retrieved content as data in the prompt, least-privilege credentials that cannot reach contacts, and a human-approval gate on the send action. The last one is the only defence that holds if the model is fully fooled.

Pitfalls and misconceptions

• Believing long context makes RAG obsolete. Windows are finite, long prompts are slow and expensive, and models use the middle of a long context poorly. Retrieval is how you construct a context the model can use.
• Starting with embeddings and skipping BM25. Term-based retrieval is nearly free, excellent on identifiers and exact phrases, and a necessary baseline; many RAG systems would be better as hybrid.
• Choosing a chunk size by intuition. It depends on the documents, the queries and the embedding model; evaluate on a labelled retrieval set.
• Evaluating only the final answer. A wrong answer may be a retrieval miss, a ranking failure or a generation error; measure recall@k, NDCG and faithfulness separately.
• Treating retrieved content as trustworthy. It is data that may be stale, wrong or adversarial; indirect prompt injection arrives through the retriever and through every knowledge tool.
• Giving the agent every tool. More tools mean harder selection, longer prompts and more attack surface; ablate and keep what moves the metric.
• Executing while planning. A bad plan then costs real actions; validate plans before executing, and gate write actions.
• Checking tool choice but not argument values. The most common agent bug is a valid tool with a plausible wrong argument; log and score parameter accuracy.
• Running agents without a step budget. A confused agent loops indefinitely; cap steps, tokens and cost per task.
• Storing raw transcripts as memory. They are unqueryable and inconsistent; extract structured, timestamped entries and define how conflicts are resolved.

Key terms

Term	Meaning
Retrieval-augmented generation (RAG)	Fetching relevant external information and adding it to the model's context before generation
Retriever / generator	The component that indexes and fetches documents / the model that answers from them
Term-based retrieval	Lexical scoring by query-term overlap (TF-IDF, BM25)
BM25	TF-IDF with term-frequency saturation and document-length normalisation; the default in Lucene and Elasticsearch
Embedding-based retrieval	Ranking by vector similarity between query and document embeddings
Approximate nearest neighbour (ANN)	Index structures that find near-neighbours fast by sacrificing a little recall (LSH, HNSW, IVF, PQ, Annoy)
HNSW	Hierarchical navigable small world graph; the dominant ANN index for its recall-latency trade-off
Product quantization (PQ)	Compressing vectors into short codes by quantising subvectors; usually paired with IVF
Context precision / recall	Fraction of retrieved chunks that are relevant / fraction of relevant chunks that were retrieved
NDCG / MRR	Rank-aware retrieval metrics: discounted graded gain normalised by the ideal ordering / mean of 1 over the rank of the first relevant hit
Hybrid retrieval	Combining term-based and embedding-based retrieval, sequentially or by rank fusion (RRF)
Reranking	Rescoring a small candidate set with a more accurate model (cross-encoder) or extra signals (recency)
Query rewriting	Transforming the user query before retrieval: resolving references, expanding, decomposing
Contextual retrieval	Prefixing each chunk with metadata or a model-written description of its place in the document before indexing
Agent	A model-based system that perceives an environment and acts on it through tools
Agent-computer interface (ACI)	A tool set and feedback format designed for a model to operate a computer (SWE-agent)
Read vs write actions	Actions that observe the environment versus actions that change it; write actions need oversight
Function calling	API mechanism where the model emits a structured tool request that the application validates and executes
Planner / evaluator / executor	Decoupled roles: propose a plan, validate it, run it
Planning granularity	Whether a plan is expressed as exact calls, natural-language steps, or a hierarchy of both
ReAct	Agent loop interleaving thought, action and observation
Reflexion	Agent framework with an evaluator and a self-reflection step whose verbal lessons feed the next attempt
Tool transition graph	Counts of which tool follows which; used to merge or drop tools
Short-term / long-term / internal memory	Context window / external store / model weights
Reflection-based memory	Extracting structured facts, preferences and lessons from an interaction instead of storing raw transcript

Further reading

• AI Engineering, Chapter 6 ("RAG and Agents"); the book's repository: https://github.com/chiphuyen/aie-book
• Chip Huyen, "Agents" (blog post adapted from the chapter): https://huyenchip.com/2025/01/07/agents.html
• Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks": https://arxiv.org/abs/2005.11401
• Chen et al., "Reading Wikipedia to Answer Open-Domain Questions" (retriever plus reader): https://arxiv.org/abs/1704.00051
• Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering": https://arxiv.org/abs/2004.04906
• Malkov and Yashunin, "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs": https://arxiv.org/abs/1603.09320
• Johnson, Douze and Jégou, "Billion-scale similarity search with GPUs" (FAISS, IVF-PQ): https://arxiv.org/abs/1702.08734
• Anthropic, "Introducing Contextual Retrieval": https://www.anthropic.com/news/contextual-retrieval
• Yang et al., "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering": https://arxiv.org/abs/2405.15793
• Schick et al., "Toolformer: Language Models Can Teach Themselves to Use Tools": https://arxiv.org/abs/2302.04761
• Lu et al., "Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models": https://arxiv.org/abs/2304.09842
• Kambhampati et al., "LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks": https://arxiv.org/abs/2402.01817
• Yao et al., "ReAct: Synergizing Reasoning and Acting in Language Models": https://arxiv.org/abs/2210.03629
• Shinn et al., "Reflexion: Language Agents with Verbal Reinforcement Learning": https://arxiv.org/abs/2303.11366
• Park et al., "Generative Agents: Interactive Simulacra of Human Behavior" (memory stream and reflection): https://arxiv.org/abs/2304.03442
• Wang et al., "Voyager: An Open-Ended Embodied Agent with Large Language Models" (skill library): https://arxiv.org/abs/2305.16291
• Robertson and Zaragoza, "The Probabilistic Relevance Framework: BM25 and Beyond" (2009)