RAG is the most asked applied topic in GenAI interviews, and it is the first thing most companies build with an LLM. Interviewers do not want the definition. They want to hear how you choose a chunk size, why you would add keyword search to vector search, what a reranker buys you, and how you debug a wrong answer. Each of those has a concrete answer, and this lesson gives you all of them with numbers you can say aloud.

The problem RAG solves

An LLM knows only what was in its training data. It has never seen your company wiki, and it does not know what changed yesterday. You have three ways to give it knowledge: put everything in the prompt (expensive, slow, and quality drops as the window fills), fine-tune it (slow to update, and it still invents details), or retrieve only the relevant passages for each question. The third is RAG. It gives you fresh answers, access control and citations, and updating knowledge is just re-indexing a document.

Embeddings and similarity

An embedding model turns a piece of text into a vector, typically 384 to 3,072 numbers. Texts with similar meaning land close together, so "time off for a new baby" sits near "parental leave policy" even though they share no words. Closeness is measured with one of three functions.

If the vectors are normalised to length 1, all three give the same ranking. Most systems normalise once at indexing time and then use the dot product.

What a vector database does

Comparing a query against every stored vector is exact but costs O(N x d) per query. That is fine up to a few hundred thousand vectors. Beyond that you need approximate nearest neighbour (ANN) search, which trades a little recall for a large speed-up. A vector database is an ANN index plus the things a real system needs: metadata filters, inserts and deletes, persistence and replication. It can be a library inside your process, an extension to a relational database, or a dedicated service.

HNSW search moving from a sparse top layer down to the dense bottom layer

HNSW (hierarchical navigable small world) is the common default. Search starts in a sparse top layer and moves down to denser layers, which gives fast lookups with high recall. Three knobs control it: M (links per node), efConstruction (build effort) and efSearch (candidates kept per query). Raising any of them improves recall and costs memory or time.

The RAG pipeline

1. Ingest: load, parse and clean documents (PDFs, HTML, tables).
2. Chunk: split into pieces, usually a few hundred tokens with some overlap. Split on structure (headings, paragraphs) where you can.
3. Embed and index: an embedding model turns each chunk into a vector, stored in a vector database with an ANN index such as HNSW. Keep a keyword index (BM25) too.
4. Retrieve: embed the query and fetch the top-k chunks by cosine similarity. Hybrid search combines dense and keyword results, often with reciprocal rank fusion.
5. Rerank: a cross-encoder scores each query-chunk pair and keeps the best few.
6. Generate: the LLM answers using only the retrieved context, with citations.
7. Evaluate: measure retrieval and generation separately.

Steps 1 to 3 run offline whenever documents change. Steps 4 to 6 run on every question.

Chunking

Chunk size is the first decision and the one with the largest effect. Small chunks retrieve precisely but lose the surrounding context. Large chunks keep context but add noise and cost. Start around 300 to 500 tokens with 10 to 20 percent overlap, then tune on a retrieval eval set. Structure-aware splitting usually beats fixed sizes: a table cut in half answers nothing. Store the heading path with each chunk ("Leave policy > Parental leave") so a paragraph still makes sense on its own.

Dense, sparse and hybrid retrieval

Dense retrieval (embeddings) matches meaning and handles paraphrases. Sparse retrieval (BM25) matches exact words such as product codes, names and error strings, which embeddings tend to blur. BM25 is the standard keyword-scoring formula: a chunk scores higher when a query word appears in it more often, and rare words count for more than common ones. It is called sparse because a text is represented by word counts over the whole vocabulary, almost all of them zero. Hybrid search runs both and is the safer default. The usual way to merge the two ranked lists is reciprocal rank fusion (RRF): each document scores the sum of 1 / (k + rank) over the lists it appears in, with k = 60. A document ranked well by both retrievers rises to the top, and no score calibration is needed.

Reranking: bi-encoder versus cross-encoder

A bi-encoder embeds the query and the document separately, so it is fast and searchable: document vectors are computed once. A cross-encoder reads the query and the document together, so it is accurate but slow, because nothing can be precomputed. Use both: retrieve about 50 candidates with the bi-encoder, then rerank them with the cross-encoder and keep the best 5 to 8.

Beyond the basic pipeline

• Query transformation: rewrite the query, generate several variants, or generate a hypothetical answer and search with that (HyDE).
• Metadata filtering: filter by user, date, department or document type before or during search. This is also how access control is enforced.
• Agentic RAG: the model decides when to search, what to search, and whether to search again, instead of one fixed retrieval step.
• GraphRAG: build a knowledge graph of entities and relations from the documents, then retrieve over the graph. It helps with questions that need facts joined across many documents.

Measuring a RAG system

Retrieval and generation fail independently, so measure them separately.

MRR (mean reciprocal rank) is the average over all questions of 1 / rank of the first relevant chunk: rank 1 scores 1, rank 4 scores 0.25. nDCG (normalised discounted cumulative gain) also counts relevant chunks further down the list, with less weight the lower they sit. Faithfulness and answer relevance are usually scored by a judge model, which splits the answer into claims and checks each one against the retrieved chunks.

RAG, fine-tuning or a long context

RAG gives the model knowledge: fresh, private or frequently changing facts. Fine-tuning changes behaviour: output format, tone, domain vocabulary, tool-call style. Start with prompting, add RAG for knowledge, and fine-tune only when you have a stable task, a real eval and roughly 500 or more good examples. They combine well. Large context windows change the design (bigger chunks, whole documents) but not the need to select, because sending everything costs more, is slower, and quality drops as the window fills.

Sizing an index. A team has 200,000 documents of about 1,500 tokens each, so 300 million tokens. With 400-token chunks and 15% overlap, each chunk adds 340 new tokens: 300M / 340 = about 882,000 chunks. (The short last chunk of each document pushes the real count toward 1 million, which does not change the conclusion.) With a 768-dimension embedding in float32, one vector is 768 x 4 = 3,072 bytes, so the vectors take 882,000 x 3,072 = 2.7 GB. The HNSW links add little at this dimension: with M = 32 the bottom layer keeps up to 64 neighbour ids x 4 bytes = 256 bytes per vector, about 0.23 GB, so plan for about 3 GB in total. That fits on one machine, so a library or a database extension is enough; a distributed vector service is not needed yet.

Fusing two rankings with RRF (k = 60). A question returns these lists:

Scores: A = 1/61 + 1/63 = 0.03227. C = 1/63 + 1/61 = 0.03227. B = 1/62 + 1/64 = 0.03175. E = 1/62 = 0.01613. D = 1/64 = 0.01563. The fused order is A and C (tied), then B, E, D. Chunks found by both retrievers beat a chunk that only one ranked highly, which is exactly the behaviour you want before the reranker makes the final call.

"""Hybrid retrieval in miniature: BM25 + cosine similarity, fused with RRF.

The "embedding" here is a toy (hashed character trigrams). It stands in for a
real embedding model so the mechanics run with NumPy alone.
"""
import math
import re
import zlib
from collections import Counter

import numpy as np

DOCS = {
    "leave-1": "Parental leave: employees get 26 weeks of paid leave after 6 months of service.",
    "leave-2": "Annual leave is 24 days per year. Up to 5 unused days carry over.",
    "it-1": "Error E-4012 means the VPN certificate expired. Renew it from the IT portal.",
    "it-2": "To reset a laptop password, open the IT portal and choose account recovery.",
    "exp-1": "Travel expenses must be filed within 30 days with receipts attached.",
}


def tokenize(text):
    return re.findall(r"[a-z0-9-]+", text.lower())


def bm25_ranking(query, docs, k1=1.5, b=0.75):
    toks = {d: tokenize(t) for d, t in docs.items()}
    avg_len = sum(len(t) for t in toks.values()) / len(toks)
    df = Counter(term for t in toks.values() for term in set(t))
    scores = {}
    for d, t in toks.items():
        tf = Counter(t)
        s = 0.0
        for term in tokenize(query):
            if term not in tf:
                continue
            idf = math.log(1 + (len(toks) - df[term] + 0.5) / (df[term] + 0.5))
            s += idf * tf[term] * (k1 + 1) / (tf[term] + k1 * (1 - b + b * len(t) / avg_len))
        if s > 0:                                  # a keyword index returns only chunks that match
            scores[d] = s
    return sorted(scores, key=scores.get, reverse=True)


def embed(text, dim=512):
    v = np.zeros(dim)
    padded = " " + " ".join(tokenize(text)) + " "
    for i in range(len(padded) - 2):
        v[zlib.crc32(padded[i:i + 3].encode()) % dim] += 1.0
    norm = np.linalg.norm(v)
    return v / norm if norm else v


def dense_ranking(query, docs):
    ids = list(docs)
    matrix = np.stack([embed(docs[d]) for d in ids])      # (N, dim), unit length
    sims = matrix @ embed(query)                          # cosine, because both are normalised
    return [ids[i] for i in np.argsort(-sims)]


def rrf(rankings, k=60):
    scores = Counter()
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores.items(), key=lambda kv: -kv[1])


if __name__ == "__main__":
    for query in ["what does error E-4012 mean", "parental leaves for employee"]:
        sparse = bm25_ranking(query, DOCS)
        dense = dense_ranking(query, DOCS)
        fused = rrf([sparse, dense])
        print("query :", query)
        print("  bm25 :", sparse[:3])
        print("  dense:", dense[:3])
        print("  fused:", [(d, round(s, 4)) for d, s in fused[:3]])
    assert rrf([["a", "b"], ["b", "a"]])[0][1] == rrf([["a", "b"], ["b", "a"]])[1][1]
    print("rrf tie check ok")