"Walk me through an LLM project you built" opens most AI engineer interviews, and the follow-ups are fixed: what went wrong, how did you evaluate it, how would you scale it. A candidate who says "Recall@5 went from 19 of 30 to 27 of 30 after I added keyword search and a reranker" is believed. A candidate who recites the definition of hybrid search is asked another question. The projects here are sized for evenings and a free or cheap GPU, and each one ends in a table of numbers, because the table is what you will be asked about.

Why three projects

Four weeks at 2 to 3 hours a day covers the whole course. Two weeks works if you skip the projects and only drill the Q&A. Skipping them costs you the third of the three rules that make the difference in interviews:

1. Say answers aloud. Reading feels like knowing. Speaking shows the gaps.
2. Always give the trade-off. "It depends on X; if A then this, if B then that" beats a one-word answer.
3. Tie every topic to one project. "I used QLoRA on a 7B model and saw..." beats any definition.

Interviewers expect your own projects in depth, with numbers for latency, cost and quality. So every project below ends in a table, not a demo video.

The loop every project follows

1. Pick a narrow problem with data you can get: your college rule book, one product's help pages, one public API.
2. Write the eval set first: about 30 cases with the expected result, before the system exists.
3. Build the simplest version and score it. That is your baseline.
4. Read the failures, change one thing, score again. Log each change with its number.
5. Record latency and cost next to quality.
6. Write the README while the numbers are fresh.

The targets below are goals for a student project on your own eval set. They are not benchmarks. Missing one and explaining why is still a good story.

Project 1: a RAG app

Build: question answering over 50 to 200 documents you care about, with citations, and a "not found" reply when the documents have no answer.

1. Ingest and chunk (start at 400 tokens with 15% overlap). Dense retrieval, top 5, answer with citations. Log the retrieved chunk ids for every answer.
2. Eval set: 30 questions with the correct chunk for each, plus 5 whose answer is not in the documents.
3. Add BM25 with RRF fusion, then a reranker over the top 50.
4. Add a faithfulness judge and check its verdicts by hand on all 30.

Project 2: a fine-tuned model

Build: a QLoRA fine-tune of a small open model for one narrow task with a checkable output, for example turning a support ticket into JSON with intent, product and urgency.

1. Collect 500 to 2,000 clean examples. Hold out 10 to 20% and never train on it.
2. Baseline: the base model with its best prompt and a few examples, scored on the held-out set.
3. Print one formatted training example and check the chat template. Train with r = 16 and alpha = 32.
4. Score the base and the fine-tuned model on the same prompts. Run a regression check on 20 general prompts.
5. Serve it on vLLM, run a load test and record TTFT (time to first token).

If the gain is not clear, do not ship, and say so in the README. "The prompt baseline reached 91% and the fine-tune 93%, so I would not pay to host it" is a strong interview answer.

Project 3: an agent with evals

Three builds prove agent skill. Do them in order, because each reuses the one before.

1. Build an agent with no framework: the loop, four tools, a step limit and tracing.
2. Build an MCP server for a real API and use it from a LangGraph agent with checkpoints and a human approval step. MCP is the open standard for exposing tools to any model client. A checkpoint is saved state a run can resume from.
3. Build an eval harness: 30 tasks, three runs each, pass rate and cost per task. Show one improvement with before and after numbers.

Grade the end state with code (was the right record changed?), not the final message.

What goes in the README

1. Problem      one sentence: who asks what, over which data
2. Run it       one command, with a sample input and output
3. Design       a diagram and five lines: what is offline, what is online
4. Eval         the dataset (size, source), the metric, the results table
5. Changes      each change with its before and after number, kept or reverted
6. Trade-offs   three decisions, what you rejected and why
7. Failures     one bug: the symptom, how you found it, the fix
8. Cost, speed  tokens per request, p95 latency, assumed prices
9. Next         what you would do with one more week

Talking about it in an interview

Prepare a two-minute answer in four parts: problem, design, trade-offs, numbers. Then prepare your own answer to these five. Each one maps to a README section.

Say each answer aloud and time it. Give counts with percentages ("27 of 30"), name the baseline, and state one trade-off before you are asked.

Rule 3 in practice: each project is the evidence for a group of topics. When a question lands on one of them, answer from the project.

From a change log to a two-minute answer. The numbers below are an invented example that shows the shape. Use your own.

A student builds a RAG app over 120 pages of college academic rules. The eval set has 30 answerable questions and 5 with no answer in the pages.

Reading the table. One question is worth 1/30 = 3.3 points. v1 to v2 is 5 questions, 16.7 points. All 5 contained course codes such as "CS-204", which dense search blurred and BM25 matched exactly. v3 to v4 changed only the prompt and a threshold, so Recall@5 did not move, as expected.

Cost per question. 5 chunks x 400 tokens = 2,000, plus 300 tokens of instructions and 50 for the question: 2,350 input tokens. The answer is 250 output tokens. At an assumed 1 USD per million input tokens and 4 USD per million output tokens: 0.00235 + 0.00100 = 0.00335 USD. At 1,000 questions a day that is 3.35 USD a day.

The spoken answer. "I built a question-answering app over 120 pages of my college's academic rules, because students keep asking the office the same questions. It chunks the pages, runs dense and keyword search, fuses the two lists, reranks 50 candidates down to 5 and answers with citations. I wrote 30 questions with the correct chunk before I built anything. Dense search alone found the right chunk for 19 of 30. Adding BM25 took that to 24, because course codes need an exact match, and a reranker took it to 27. The trade-off was latency: the reranker added time to every query, which I accepted because nobody needs this answer in under a second. Faithfulness is 27 of 30 by a judge I checked by hand, and it refuses all 5 unanswerable questions. It costs about a third of a cent per question at the prices I assumed."

That is problem, design, trade-off and numbers in about 145 words, with a baseline named and every claim countable.

"""A project eval harness: score two versions of a system on the same cases
and print a before/after table you can paste into a README.

The predictions are hard-coded so this runs anywhere. In a real project they
come from running your RAG app, fine-tuned model or agent on each case.
"""
import json
import math
import re

CASES = [
    {"id": "q01", "tag": "lookup", "scorer": "exact", "expected": "30 days"},
    {"id": "q02", "tag": "lookup", "scorer": "exact", "expected": "E-4012"},
    {"id": "q03", "tag": "lookup", "scorer": "number", "expected": 26},
    {"id": "q04", "tag": "lookup", "scorer": "number", "expected": 1499.0},
    {"id": "q05", "tag": "multi-fact", "scorer": "contains", "expected": ["24 days", "5 days"]},
    {"id": "q06", "tag": "multi-fact", "scorer": "contains", "expected": ["receipt", "30 days"]},
    {"id": "q07", "tag": "multi-fact", "scorer": "contains", "expected": ["IT portal", "certificate"]},
    {"id": "q08", "tag": "extract", "scorer": "json", "expected": {"city": "Pune", "pincode": "411001"}},
    {"id": "q09", "tag": "extract", "scorer": "json", "expected": {"city": "Kochi", "pincode": "682001"}},
    {"id": "q10", "tag": "no-answer", "scorer": "refusal", "expected": None},
    {"id": "q11", "tag": "no-answer", "scorer": "refusal", "expected": None},
    {"id": "q12", "tag": "no-answer", "scorer": "refusal", "expected": None},
]

V1 = {  # baseline: dense retrieval only
    "q01": "30 days.", "q02": "The error code is E-4021", "q03": "You get 26 weeks of paid leave.",
    "q04": "Rs 1,499", "q05": "Annual leave is 24 days.", "q06": "File within 30 days with a receipt.",
    "q07": "Renew it.", "q08": '{"city": "Pune", "pincode": "411001"}',
    "q09": 'Here is the JSON: {"city": "Kochi"}', "q10": "The gym opens at 6 am.",
    "q11": "I could not find this in the documents.", "q12": "Probably yes.",
}
V2 = {  # candidate: hybrid retrieval, a reranker and a "say not found" instruction
    "q01": "30 days", "q02": "E-4012", "q03": "26 weeks", "q04": "The refund is 1499 rupees.",
    "q05": "24 days a year, and up to 5 days carry over.", "q06": "Within 30 days, receipts attached.",
    "q07": "The VPN certificate expired. Renew it from the IT portal.",
    "q08": '{"city": "Pune", "pincode": 411001}', "q09": '{"city": "Kochi", "pincode": "682001"}',
    "q10": "Not found in the documents.", "q11": "I could not find this in the documents.",
    "q12": "Not found in the documents.",
}


def norm(text):
    return re.sub(r"[^a-z0-9 -]", "", text.lower()).strip()


def score(case, output):
    kind, expected = case["scorer"], case["expected"]
    if kind == "exact":
        return norm(output) == norm(expected)
    if kind == "contains":                      # every required fact must appear
        return all(norm(fact) in norm(output) for fact in expected)
    if kind == "number":                        # first number in the output, 1% tolerance
        found = re.search(r"-?\d+(?:\.\d+)?", output.replace(",", ""))
        return bool(found) and math.isclose(float(found.group()), expected, rel_tol=0.01)
    if kind == "json":                          # must parse, and every field must match exactly
        try:
            return json.loads(output) == expected
        except json.JSONDecodeError:
            return False
    if kind == "refusal":                       # the right answer is to say it is not there
        return any(p in output.lower() for p in ("not found", "could not find", "do not know"))
    raise ValueError(f"unknown scorer: {kind}")


def run(predictions):
    return {c["id"]: score(c, predictions[c["id"]]) for c in CASES}


if __name__ == "__main__":
    r1, r2 = run(V1), run(V2)
    n = len(CASES)
    print("| Slice | Cases | v1 baseline | v2 candidate |")
    print("| --- | --- | --- | --- |")
    for tag in dict.fromkeys(c["tag"] for c in CASES):
        ids = [c["id"] for c in CASES if c["tag"] == tag]
        print(f"| {tag} | {len(ids)} | {sum(r1[i] for i in ids)}/{len(ids)} | {sum(r2[i] for i in ids)}/{len(ids)} |")
    s1, s2 = sum(r1.values()), sum(r2.values())
    print(f"| all | {n} | {s1}/{n} ({s1 / n:.0%}) | {s2}/{n} ({s2 / n:.0%}) |")

    fixed = [i for i in r1 if not r1[i] and r2[i]]
    broke = [i for i in r1 if r1[i] and not r2[i]]
    print(f"\nfixed by v2 : {len(fixed)} {fixed}")
    print(f"broken by v2: {len(broke)} {broke}   <- read these before you ship")
    print(f"net change  : {len(fixed) - len(broke):+d} of {n} cases")
    print(f"one case is worth {1 / n:.1%}, so report counts, not only percentages")

    assert score(CASES[3], "Rs 1,499") and not score(CASES[7], '{"city": "Pune", "pincode": 411001}')
    assert not score(CASES[1], "The error code is E-4021")