Senior interviews go one question deeper than the definition. After "how do you get JSON out of a model?" comes "how does the server guarantee it?". After "what is an agent?" comes "what is in the code around the model?". These ten topics are where those second questions come from. They are also the levers you pull in real work: most of them trade cost, latency or reliability against quality. You do not need all ten at once. Use the table to pick the two or three that match the system you are building, and learn those properly.

The map

Reasoning models and adaptive reasoning

Reasoning models think before answering: they write reasoning tokens first, then the answer. Training adds reinforcement learning with verifiable rewards. The focus has moved from raw ability to efficiency: effort adjusts to how hard the prompt is.

Reasoning tokens are generated like any other output tokens. You pay for them and you wait for them before the answer starts. So treat effort as a setting per task: low for extraction and routing, high for planning and debugging. A line to say aloud: "I set the thinking level per task, because reasoning tokens cost latency and money." These models need a clear goal more than a step-by-step script. Learn first: run one eval set at two effort levels and compare pass rate, latency and tokens.

Mixture of experts (MoE)

The feed-forward layer is split into many expert networks, and a router sends each token to only a few of them. The model has many parameters but uses a small part per token, so quality rises without the same rise in compute. Many recent open-weight models use sparse MoE designs.

A mixture-of-experts router sending one token to 2 of 8 expert networks

Here E_i is expert i and g_i is the weight the router gives it. For a builder the point is two different numbers. Speed follows the active parameters, the experts that run. Memory follows the total parameters, because every expert stays loaded. Training adds a load-balancing loss so the router does not send everything to the same few experts. Learn first: top-k routing, and total versus active parameters.

Distillation

A large teacher model generates outputs and a small student model is trained to match them. This is how small models get close to large-model quality on a narrow task. The classic form matches the teacher's whole probability distribution, softened with a temperature. These soft targets carry more signal than one label: a teacher that gives 70% to one answer and 25% to another shows the student which wrong answer is nearly right. With an API teacher you only see text, so you fine-tune the student on the teacher's answers.

It is the standard move when a large model works but costs too much at volume. The risk: the student copies the teacher's mistakes, so filter the teacher's outputs and evaluate on human-checked examples. Learn first: generate 1,000 to 2,000 teacher answers for one task and fine-tune a small model on them.

Constrained (structured) decoding

At each step the model gives a score to every token in the vocabulary. In constrained decoding, a grammar or JSON schema is compiled into a state machine that knows which tokens are legal next. Every other token is masked (its score is set to minus infinity) before sampling, so invalid tokens cannot be sampled.

This guarantees the form, not the content: the output parses and matches the schema, but a value can still be wrong. Output can also be cut off by the token limit. So validate in code, and retry with the error message if validation fails.

Two practical points. The mask is needed at every step over a vocabulary of tens of thousands of tokens or more, so engines compile the grammar once and reuse it. And the schema shapes quality: if a field for the reasoning comes after the answer field, the model must answer before it has thought, so put reasoning fields first. Learn first: the structured output option of your provider or serving engine. SGLang is known for fast constrained decoding.

Speculative decoding

A small draft model proposes several tokens and the large model checks them in one pass. Accepted tokens are free speed, and the output distribution stays the same, because the large model verifies every drafted token and rejects the ones it would not have produced.

The rule that makes this exact: a drafted token x is accepted with probability min(1, p(x) / q(x)), where p is the large model's probability and q is the draft's. On the first rejection, the large model samples a replacement from a corrected distribution and the rest of the draft is dropped. The large model needs no retraining.

You only gain when the draft guesses right often, so it works best on predictable text such as code and boilerplate. The price is a second model in memory. Learn first: the acceptance rate and the expected-tokens formula in the worked example.

Long-running and persistent agents

These are agents that work for hours and keep state. Reliability and security are the open problems. Errors compound over long tasks, so add checkpoints, limits and approvals. The state must live outside the context window: a progress file the agent re-reads, saved checkpoints, and a memory store. Then a run can stop, resume a day later, or survive a crash. More hours also means more untrusted content read, so the injection risk grows with run length. Learn first: checkpoint and resume, idempotent writes (a retry has no extra effect), and budgets for steps, tokens, time and money.

Harness engineering

The harness is everything around the model: the loop, the tools, context management, permissions and checks. With the same model, a better harness often finishes more tasks. Four parts to know:

• Skills: packaged instructions, a folder with an instruction file and sometimes scripts, that the agent loads when a task needs them. Until then only a short description sits in the context.
• Hooks: your own checks that run before or after a tool call. A before-hook can block a call, such as a delete outside the project folder. An after-hook can run tests. The rule lives in code, not in the prompt.
• Sub-agents: a sub-agent explores in its own window and returns a short summary, so the lead agent's context stays clean. The cost is more total tokens, more latency if they run in sequence, and lost detail when results are summarised. They pay off when sub-tasks are independent and large.
• Compaction: when the window is nearly full, summarise the history and keep decisions, open items and key facts.

Learn first: write the agent loop yourself, then add one hook and compaction.

Open-weight models closing the gap

Open models are close to closed ones on many benchmarks, often with sparse MoE designs and long context. Self-hosting is now a real option when cost, privacy or control matter. You take on the serving work in exchange: GPUs, autoscaling, upgrades and your own evals. Learn first: serve one open model and compare it with a hosted model on your eval set.

Small fine-tuned model plus RAG

A QLoRA-tuned small open model with good retrieval is a common cost-for-quality choice. Fine-tuning teaches the behaviour (format, tone, tool-call style) and retrieval supplies the facts. The usual split: a large model for hard reasoning and a small tuned one for high-volume tasks, with a router between them. Learn first: the router and the eval that proves the small model is good enough.

GraphRAG

Build a knowledge graph of entities and relations from the documents, then retrieve over the graph. It helps with questions that need facts joined across many documents, such as "which suppliers are shared by our three largest customers?". One published design also writes a summary in advance for each cluster of closely related entities, which answers corpus-wide questions such as "what are the main themes?". The cost is at indexing: a model must read every chunk to extract entities and relations, and again when documents change. Learn first: plain hybrid RAG. Add a graph only when your eval shows multi-document questions failing.

Is speculative decoding worth it? A draft model proposes 4 tokens per round (call this g = 4). The large model accepts each drafted token with probability a, and stops at the first rejection. Every round also yields one token from the large model itself: the correction after a rejection, or a bonus token when all four are accepted. So the expected number of tokens per round is

With a = 0.8: E = (1 - 0.8^5) / 0.2 = (1 - 0.328) / 0.2 = 3.36 tokens per round, against 1 token per pass without drafting.

The draft is not free. Assume one draft step costs 5% of a large-model pass. A round costs 4 x 0.05 + 1 = 1.2 large passes. Speed-up = 3.36 / 1.2 = 2.8 times.

The method pays off only when the draft is right most of the time. Measure the acceptance rate on your own traffic before you deploy it.

MoE sizing. Take a layer with width 4,096 and a feed-forward hidden size of 16,384. One expert has 2 x 4,096 x 16,384 = 134 million weights. Eight experts hold 1.07 billion weights in this layer, and all of them use GPU memory. With top-2 routing, a token passes through 268 million of them, which is 25% of the feed-forward compute. Size memory from the total and speed from the active part.

"""Constrained decoding in miniature.

A tiny state machine tracks where we are inside a JSON object with the schema
{"city": string, "temp": number}. At every step it lists the tokens that keep
the output valid, masks every other token to minus infinity, and picks the
best token that is left. The model scores are scripted, so this runs anywhere.
"""
import json
import math

VOCAB = ["Sure", "!", "{", "}", '"city"', '"temp"', ":", ",", '"Pune"', '"hot"', "31", "<eos>"]
SCHEMA = {'"city"': "string", '"temp"': "number"}       # every key is required
STRINGS = {'"Pune"', '"hot"'}
NUMBERS = {"31"}


def allowed_tokens(state):
    """Return the set of tokens that keep the output inside the grammar."""
    phase, used, key = state["phase"], state["used"], state["key"]
    if phase == "start":
        return {"{"}
    if phase == "key":                                   # a key not used yet
        return set(SCHEMA) - used
    if phase == "colon":
        return {":"}
    if phase == "value":                                 # type comes from the schema
        return STRINGS if SCHEMA[key] == "string" else NUMBERS
    if phase == "after_value":                           # "}" only when all keys are present
        return {"}"} if used == set(SCHEMA) else {","}
    if phase == "end":
        return {"<eos>"}
    return set()


def advance(state, token):
    """Move the state machine forward by one emitted token."""
    phase = state["phase"]
    if phase == "start":
        state["phase"] = "key"
    elif phase == "key":
        state["key"] = token
        state["used"] = state["used"] | {token}
        state["phase"] = "colon"
    elif phase == "colon":
        state["phase"] = "value"
    elif phase == "value":
        state["phase"] = "after_value"
    elif phase == "after_value":
        state["phase"] = "end" if token == "}" else "key"
    elif phase == "end":
        state["phase"] = "done"


# Scripted "model" scores (logits) for each step. The top choice is often a
# token that would break the JSON, which is what real models sometimes do.
# A real model would react to its earlier tokens; this script does not.
SCRIPT = [
    {"Sure": 3.1, "{": 2.4, '"city"': 0.3},              # wants to chat first
    {'"temp"': 1.9, '"city"': 1.7, "}": 2.2},            # wants to close early
    {":": 4.0, ",": 0.5},
    {"31": 2.0, '"hot"': 1.5, "}": 0.2},                 # schema says number here
    {"}": 2.6, ",": 1.1, "!": 0.4},                      # "city" is still missing
    {'"temp"': 2.8, '"city"': 1.0},                      # would repeat a key
    {":": 3.5},
    {"31": 2.2, '"Pune"': 2.1, '"hot"': 0.9},            # schema says string here
    {",": 1.8, "}": 1.6, "!": 1.0},
    {"!": 2.0, "<eos>": 1.9},
]


def decode(constrained):
    state = {"phase": "start", "used": set(), "key": None}
    out = []
    for step, scores in enumerate(SCRIPT, start=1):
        logits = {tok: scores.get(tok, -5.0) for tok in VOCAB}     # unlisted tokens score low
        free_pick = max(logits, key=logits.get)
        if not constrained:
            out.append(free_pick)
            continue
        allowed = allowed_tokens(state)
        masked = {tok: (s if tok in allowed else -math.inf) for tok, s in logits.items()}
        pick = max(masked, key=masked.get)
        note = "" if pick == free_pick else f"   (model wanted {free_pick})"
        print(f"step {step:>2}  allowed = {sorted(allowed)}  ->  pick {pick}{note}")
        out.append(pick)
        advance(state, pick)
        if state["phase"] == "done":
            break
    return out


if __name__ == "__main__":
    free = decode(constrained=False)
    print("unconstrained:", " ".join(free))
    try:
        json.loads("".join(t for t in free if t != "<eos>"))
    except json.JSONDecodeError as e:
        print("  json.loads fails:", e.msg, "\n")

    tokens = decode(constrained=True)
    text = "".join(t for t in tokens if t != "<eos>")
    parsed = json.loads(text)
    print("\nconstrained  :", text)
    assert parsed == {"temp": 31, "city": "Pune"}
    assert set(parsed) == {"city", "temp"} and isinstance(parsed["temp"], int)
    print("parses, has both keys, types match the schema")