This round decides the level you are hired at. The interviewer is not checking whether you know what RAG or vLLM is. They are checking whether you can turn a vague request into numbers, pick a model path and defend it, and say what breaks at 10 times the traffic. A fixed structure lets you spend your 45 minutes on the hard part and not on deciding what to draw. The same structure works at work: every design review for an LLM feature asks about traffic, tokens, latency, cost and evals.

Most LLM design questions use the same five layers. Draw them first, then go deep where the interviewer pushes.

The five layers

Five stacked layers of an LLM system with a data-and-tools box beside orchestration and an observability bar underneath

Observability and evals sit under every layer: traces, latency, cost and quality scores.

A request flows top to bottom; the orchestration layer is where RAG, agents and guardrails live, and it is where most of the design discussion happens. Orchestration is the only layer that talks to the data-and-tools box, and the link goes both ways: it reads chunks and memories, calls tools, and writes new memories back. If you use a hosted API, the provider runs the bottom two layers. You still own the top three, the data and the observability.

The answer framework

1. Clarify: users, traffic, tokens per request, latency target, quality bar, data sensitivity, budget.
2. Choose the model path: hosted API or self-hosted open model, and why.
3. Draw the layers above and name what sits in each.
4. Go deep on the hard part: retrieval quality, agent reliability or serving capacity, depending on the question.
5. Estimate capacity with simple arithmetic.
6. Add evals and monitoring: what you measure offline and online.
7. Cover failure and cost: fallbacks, rate limits, caching, guardrails.

A workable split for a 45-minute round: clarify 5 minutes, model path 3, layers 5, the hard part 20, capacity 5, evals 4, failure and cost 3. Steps 1 to 3 are setup. Most of the time belongs to step 4.

Step 1: what to ask, and why it changes the design

If the interviewer gives no numbers, state your own and move on: "I will assume 50 requests per second at peak and 500 output tokens."

Step 2: hosted API or self-hosted

A common answer is both: launch on a hosted API, then move the high-volume, simple requests to a small self-hosted model and keep the hosted model for hard requests and as a fallback.

Step 4: find the hard part

If you are not sure which one the interviewer wants, ask: "Should I go deeper on retrieval or on serving?"

Step 5: capacity arithmetic

A replica is one copy of the model server with its own GPU or GPUs. Headroom is spare capacity kept for spikes and for a replica that fails.

Capacity example (illustrative numbers): peak load is 50 requests per second with 500 output tokens each, so you need 25,000 output tokens per second. If one replica sustains 2,500 tokens per second at your latency target, you need 10 replicas, or 13 with 30% headroom. Say that you would measure the per-replica number with a load test.

The per-replica figure is not a constant: it depends on model size, the GPU, the batch size, prompt length and the latency target. The inference lesson's example, one 7B replica with 32 requests in the batch at 40 ms per token, produces 32 / 0.040 = 800 tokens per second, while 2,500 assumes a replica that holds about 75 streams at 30 ms per token. Both are assumptions, and only a load test on your model, your GPU and your prompts gives the number to use.

Two more formulas help. Concurrent streams = requests per second x seconds each response stays open. This converts "1,000 concurrent users" into a rate. Daily tokens = requests per day x tokens per request, which feeds the cost estimate. Size for the peak, not the average: in the worked example below the peak is 3.5 times the daily mean.

Cost

Here p is the price per million tokens. At an assumed 0.50 USD per million input tokens and 1.50 USD per million output tokens, a request with 1,000 input and 500 output tokens costs (1,000 x 0.50 + 500 x 1.50) / 1,000,000 = 0.00125 USD.

For self-hosting, monthly cost = replicas x GPUs per replica x GPU-hour price x 720 hours. At an assumed 2.50 USD per GPU-hour, the 13 replicas above cost 13 x 2.50 x 720 = 23,400 USD a month. A GPU costs the same whether busy or idle, so utilisation decides the price per token: at 2,500 tokens per second, that GPU makes a million output tokens for 0.28 USD when always busy and 2.78 USD when 10% busy.

Latency budget

Split the target across stages before choosing components. Here is an illustrative budget for a RAG chat whose p95 time to first token (TTFT) must be 1 second, meaning 95% of requests show their first token within that time:

After the first token, time per output token takes over: 300 tokens at 30 ms each is another 9 seconds. Streaming is what makes that acceptable. Every extra model call before the first token (query rewriting, an LLM guardrail, an agent step) costs hundreds of milliseconds, so name the ones you can afford.

Where caching goes

Fallbacks and rate limits

• Timeouts on time to first token and on total time, so a stuck call fails fast.
• Retries with exponential backoff and jitter, only for errors that can succeed later (429, 5xx).
• Fallback chain: a second provider or model, then a smaller model, then a cached or fixed answer. Say that quality drops and how you would tell the user.
• Rate limits in tokens per minute, per user and per tenant, not only in requests. One request can cost 100 times another.
• Backpressure: when the queue is too long to meet the latency target, reject with a clear error; do not let every request get slow.
• Guardrails on input and output, and a cap on output tokens and agent steps.

Evals and monitoring

Offline, before a change ships: a fixed eval set of real questions, task metrics, an LLM judge checked against human labels, and a gate that blocks a release when scores drop. Online, after it ships: p95 time to first token, time per output token, error and fallback rates, tokens and cost per request, cache hit rate, guardrail triggers, user feedback and the task metric (resolved tickets, accepted suggestions). Trace every request with one span per stage, and feed failed production cases back into the eval set.

Common questions

The serving question asked most often

"Design an inference batching system for one GPU while users wait" is reported again and again at AI labs. The expected answer: requests enter a queue, and a scheduler fires a batch when either the maximum batch size or a short maximum wait window is reached. Call the wait window your latency-throughput dial. Group requests by length so padding does not dominate, add backpressure when the queue grows, and mention continuous batching for token generation.

A support assistant for a shopping app. Assumptions: 2,000,000 requests a day, a peak of 80 requests per second, 1,500 input tokens per request (system prompt, retrieved chunks, history) and 300 output tokens.

1. Check the peak. The average is 2,000,000 / 86,400 = 23.1 requests per second, so the peak is about 3.5 times the average. Size for 80.
2. Tokens per second. 80 x 300 = 24,000 output tokens per second at peak.
3. Replicas. Assume one replica sustains 3,000 output tokens per second while p95 TTFT stays under 1 second, a number to confirm with a load test using these 1,500-token prompts. 24,000 / 3,000 = 8 replicas.
4. Headroom. 8 x 1.3 = 10.4, so 11 replicas.
5. Cross-check with concurrency. With a 0.5 s TTFT and 30 ms per token, one response stays open for 0.5 + 299 x 0.03 = 9.47 s. Concurrent streams = 80 x 9.47 = 758, or 69 per replica across 11. At 30 ms per token a replica producing 3,000 tokens per second carries 90 streams, so 69 fits.
6. Daily tokens. Input: 2,000,000 x 1,500 = 3.0 billion. Output: 2,000,000 x 300 = 0.6 billion.
7. Hosted cost. At an assumed 0.50 USD per million input tokens and 1.50 USD per million output tokens: 3,000 x 0.50 + 600 x 1.50 = 1,500 + 900 = 2,400 USD a day, 72,000 USD a month, 0.0012 USD per request.
8. Self-hosted cost. At an assumed 2.50 USD per GPU-hour with one GPU per replica, always on: 11 x 2.50 x 720 = 19,800 USD a month, 0.00033 USD per request.
9. Decision. On these assumptions hosting is 3.6 times the GPU bill. That gap has to pay for the engineers who run the serving stack, and it only holds if a small open model meets the quality bar. Scaling down at night widens the gap. If traffic were 10 times lower, the hosted bill would be 7,200 USD a month against 3,600 USD for 2 replicas, and a 3,600 USD saving does not pay for running a serving stack.

Say the last step aloud. The interviewer wants the decision and what would change it, not only the numbers.

"""Capacity and cost calculator for an LLM service. Pure Python.

Every price and every per-replica throughput below is an assumption.
Replace them with your provider's price list and your own load test.
"""
import math

HOURS_PER_MONTH = 24 * 30


def capacity(peak_rps, output_tokens, replica_tps, headroom=0.30):
    """Replicas needed at peak. replica_tps = output tokens per second that one
    replica sustains at the latency target (measure it with a load test)."""
    need_tps = peak_rps * output_tokens
    base = need_tps / replica_tps
    with_headroom = math.ceil(round(base * (1 + headroom), 9))
    return {"need_tps": need_tps, "base": base, "replicas": with_headroom}


def concurrent_streams(peak_rps, output_tokens, ttft_s, tpot_s):
    """Streams open at once = arrival rate x time each request stays open."""
    duration = ttft_s + (output_tokens - 1) * tpot_s
    return duration, peak_rps * duration


def hosted_cost(requests_per_day, input_tokens, output_tokens, usd_in_per_m, usd_out_per_m):
    per_request = (input_tokens * usd_in_per_m + output_tokens * usd_out_per_m) / 1e6
    return {"per_request": per_request, "per_day": per_request * requests_per_day,
            "per_month": per_request * requests_per_day * 30}


def self_hosted_cost(replicas, usd_per_gpu_hour, requests_per_day, gpus_per_replica=1):
    per_month = replicas * gpus_per_replica * usd_per_gpu_hour * HOURS_PER_MONTH
    return {"per_month": per_month, "per_request": per_month / (requests_per_day * 30)}


def estimate(name, peak_rps, requests_per_day, input_tokens, output_tokens, replica_tps,
             ttft_s, tpot_s, usd_in_per_m, usd_out_per_m, usd_per_gpu_hour):
    cap = capacity(peak_rps, output_tokens, replica_tps)
    duration, streams = concurrent_streams(peak_rps, output_tokens, ttft_s, tpot_s)
    api = hosted_cost(requests_per_day, input_tokens, output_tokens, usd_in_per_m, usd_out_per_m)
    own = self_hosted_cost(cap["replicas"], usd_per_gpu_hour, requests_per_day)
    print(name)
    print(f"  output tokens per second at peak : {peak_rps} x {output_tokens} = {cap['need_tps']:,}")
    print(f"  replicas before headroom         : {cap['need_tps']:,} / {replica_tps:,} = {cap['base']:.1f}")
    print(f"  replicas with 30% headroom       : {cap['base'] * 1.3:.1f} -> {cap['replicas']}")
    print(f"  one response stays open          : {duration:.2f} s, so {streams:.0f} concurrent streams,"
          f" {streams / cap['replicas']:.0f} per replica")
    print(f"  tokens per day                   : {requests_per_day * input_tokens / 1e9:.2f} B input,"
          f" {requests_per_day * output_tokens / 1e9:.2f} B output")
    print(f"  hosted API (assumed prices)      : {api['per_request']:.5f} USD per request,"
          f" {api['per_day']:,.0f} USD per day, {api['per_month']:,.0f} USD per month")
    print(f"  self-hosted (assumed GPU price)  : {own['per_month']:,.0f} USD per month,"
          f" {own['per_request']:.5f} USD per request")
    print(f"  hosted / self-hosted             : {api['per_month'] / own['per_month']:.1f}x, before engineering cost")
    return cap, api, own


if __name__ == "__main__":
    # 1. The guide's example: 50 requests per second, 500 output tokens,
    #    2,500 tokens per second per replica, 30% headroom -> 13 replicas.
    #    Requests per day, input tokens and prices are extra assumptions for the cost lines.
    cap, api, own = estimate("Guide example", peak_rps=50, requests_per_day=1_500_000,
                             input_tokens=1_000, output_tokens=500, replica_tps=2_500,
                             ttft_s=0.5, tpot_s=0.03, usd_in_per_m=0.50, usd_out_per_m=1.50,
                             usd_per_gpu_hour=2.50)
    assert (cap["need_tps"], cap["base"], cap["replicas"]) == (25_000, 10.0, 13)
    assert abs(api["per_request"] - 0.00125) < 1e-12 and round(own["per_month"]) == 23_400

    # 2. The worked example: shopping-app support assistant.
    cap, api, own = estimate("Worked example", peak_rps=80, requests_per_day=2_000_000,
                             input_tokens=1_500, output_tokens=300, replica_tps=3_000,
                             ttft_s=0.5, tpot_s=0.03, usd_in_per_m=0.50, usd_out_per_m=1.50,
                             usd_per_gpu_hour=2.50)
    assert (cap["need_tps"], cap["base"], cap["replicas"]) == (24_000, 8.0, 11)
    assert round(api["per_month"]) == 72_000 and round(own["per_month"]) == 19_800

    # 3. Cost per million output tokens on your own GPU depends on utilisation.
    print("Self-hosted cost per million output tokens at an assumed 2.50 USD per GPU-hour, 2,500 tokens/s:")
    for utilisation in (1.0, 0.5, 0.1):
        usd = 2.50 / 3600 / (2_500 * utilisation) * 1e6
        print(f"  GPU {utilisation:>4.0%} busy: {usd:.2f} USD")
    print("all checks passed")