Serving questions test whether you can reason with numbers. "Serve an open model to 1,000 users" and "design a batching system for one GPU" are asked at AI labs and platform teams, and the follow-ups are always the same: where does the memory go, why is the first token slow, what happens to latency when you raise the batch size. In real work the same ideas decide your GPU bill. A team that understands the KV cache, continuous batching and prefix caching can serve the same traffic on a fraction of the hardware.

Serving comes down to three numbers: latency, throughput and cost per token. Almost every technique here trades one for another, and saying which trade you are making is what interviewers listen for.

Two phases: prefill and decode

• Prefill processes the whole prompt in one parallel pass and saves what it computed for every prompt token. Prefill sets time to first token.
• Decode then generates one token at a time. Each step feeds in only the newest token. Decode sets tokens per second.

Prompt: "The capital of France is"   (5 tokens)

Prefill   [The][capital][of][France][is]   one parallel pass      -> " Paris"   cache: 5 tokens
Decode 1  [ Paris]                         reads 5 cached tokens  -> "."        cache: 6 tokens
Decode 2  [.]                              reads 6 cached tokens  -> <eos>      cache: 7 tokens

Prefill can be parallel because every prompt token is already known. Decode cannot: token 2 of the answer depends on token 1. So a 2,000-token prompt costs one pass, and a 300-token answer costs 300 passes, one after another. That is why capping output length cuts total latency more than trimming the prompt does.

Each decode pass reads every weight of the model to produce a single token. If 32 requests share the pass, the weights are read once and 32 tokens come out. This is the reason batching helps decode so much.

The KV cache

Attention compares the new token's query with a key and a value from every earlier token. A decoder token only looks backwards, so those keys and values never change once computed. During decode, the keys and values of all earlier tokens are stored for every layer, so each new token only computes its own. Without the cache, every step would push the full sequence through the model again. For a 2,000-token prompt and a 300-token answer that is 2,000 + 2,001 + ... + 2,299 = 644,850 token positions through the layers. With the cache it is 2,299: the prompt once, then one position per step. The cache makes generation fast, and it is the main memory cost besides the weights.

The 2 is for keys plus values. Heads x head dim equals the hidden size in a standard model. Models with grouped-query attention share keys and values between heads, so count the KV heads only. The cache is not shared between requests: every concurrent request holds its own.

Memory maths you should be able to do

Weights = parameters x bytes per parameter. A 7B model in FP16 (2 bytes) needs about 14 GB. In 4-bit it needs about 3.5 GB. A 70B model in FP16 needs about 140 GB. Then add the KV cache, which grows with layers x sequence length x batch size.

Quantization stores weights in fewer bits (FP16 to INT8 to 4-bit). Memory drops and speed rises, with a small quality loss.

Names to know:

• GPTQ and AWQ are post-training methods: they quantize an already trained model using a small calibration set, with no retraining. GPTQ goes layer by layer and adjusts the remaining weights to cancel each rounding error. AWQ looks at activations to find the small share of weights that matter most and protects them.
• GGUF is the file format llama.cpp uses. One file holds the quantized weights, the tokenizer and the metadata, at a bit width you choose when you download it.
• NF4 is the 4-bit data type used by QLoRA.

GPU engines such as vLLM load a GPTQ or AWQ checkpoint; llama.cpp and Ollama load GGUF; TensorRT-LLM compiles the model into an engine file built for one GPU type. Quantizing the weights does not shrink the KV cache, so long contexts stay expensive.

The metrics

• TTFT (time to first token): queue wait plus prefill.
• TPOT (time per output token): the gap between streamed tokens.
• Tokens per second: 1 / TPOT for one request.
• End-to-end latency: from request to last token.
• Throughput: total tokens per second across all requests on the server.

N is the number of output tokens. With streaming, users feel TTFT first and TPOT after that, so a chat product sets a target for each. A batch job cares only about throughput. Always quote p95 or p99, not the average. p95 is the latency that 95% of requests beat. Latency has a long tail, and the slowest 5% of requests are the ones users complain about.

Batching: static and continuous

A GPU can run many requests in one forward pass for little extra time, so batching is where throughput comes from.

Static batching waits to fill a batch and holds every request until the longest finishes. A request that needs 2 tokens sits in a batch with one that needs 8, and its slot does nothing for 6 steps.

Continuous batching adds new requests and removes finished ones at every decode step, so the GPU stays busy.

Two timelines of six requests on three GPU slots: static batching leaves slots idle, continuous batching refills them

In the picture, six requests need 2, 8, 4, 3, 5 and 2 tokens on three slots. Static batching finishes at step 13 with the GPU 61.5% busy. Continuous batching finishes at step 9 with the GPU 88.9% busy, and the average wait drops from 4.0 steps to 1.8. The gap widens when output lengths vary more, which is the normal case for chat.

What limits the batch is memory first and latency second. Every running request needs room for its KV cache, and every extra request slows each step a little.

The one-GPU batching question. "Design an inference batching system for one GPU while users wait" is reported again and again at AI labs. The expected answer: requests enter a queue, and a scheduler fires a batch when either the maximum batch size or a short maximum wait window is reached. Call the wait window your latency-throughput dial. Group requests by length so padding does not dominate, add backpressure when the queue grows, and mention continuous batching for token generation. Padding is the filler that brings every prompt in a batch up to the longest one. Backpressure means the server rejects or delays new requests once the queue is too long.

PagedAttention and prefix caching

A request's cache grows one token at a time, and the server cannot know the final length. Older engines reserved one contiguous block per request, sized for the longest possible output. Most of that reservation stayed empty, and the gaps between blocks were too small to reuse.

PagedAttention stores the KV cache in small fixed-size blocks, like virtual memory pages, instead of one contiguous block per request. A block is handed out only when the sequence needs it, and a table maps each request to its blocks. Less fragmentation means bigger batches on the same GPU.

Prefix caching reuses the KV cache for a shared prompt prefix (system prompt, few-shot examples, chat history) across requests. The second request with the same first 1,500 tokens skips prefill for them, so TTFT drops. The match must be exact from the first token, so put stable text first and anything that changes (the user message, the time) last.

Speculative decoding, FlashAttention and multi-GPU

• Speculative decoding: a small draft model proposes several tokens and the large model checks them in one pass. Accepted tokens are free speed; the output distribution stays the same. Say the draft proposes 4 tokens and the large model agrees with the first 3. It keeps those, replaces the fourth with its own choice, and has produced 4 tokens for the price of one large pass plus four cheap ones. If the first guess is wrong, it still gets 1 token, so the output is never worse, only the speed-up is lost. The method spends extra compute to cut latency, so it helps most when the GPU has spare capacity.
• FlashAttention: computes exact attention in a memory-efficient way on the GPU, by working in blocks that fit in fast on-chip memory. Same result, less memory, faster.
• Multi-GPU: tensor parallelism splits each layer across GPUs when a model does not fit on one. A 70B model at 140 GB in FP16 does not fit on one 80 GB GPU.

Serving engines

To serve many fine-tuned variants, keep one base model in memory and load a LoRA adapter (a small set of extra weights from fine-tuning) per request. This is multi-LoRA serving.

Autoscaling and load testing

Autoscaling. Scale on queue depth, tokens in flight or TTFT, not CPU.

Model loading is slow: a new replica must start a node, download tens of gigabytes of weights and load them into GPU memory. So keep a warm minimum of replicas, scale up early and scale down slowly. Put a gateway in front for rate limits, retries and fallback to another model.

Load testing. Replay prompts with a realistic length distribution, sweep concurrency upward, and record TTFT and TPOT percentiles at each level. The point where latency climbs sharply is your capacity per replica. Send requests at a fixed arrival rate as well as with a fixed number of simulated users: simulated users who wait for each reply slow down when the server does, which hides the queue that real traffic would build. Use the measured number in capacity planning, not a number from a blog post.

KV cache for a 7B model. Take 32 layers, 32 heads, head dimension 128 (hidden size 4,096) and FP16 values (2 bytes).

1. Per token: 2 x 32 x 32 x 128 x 2 bytes = 524,288 bytes, about 0.52 MB.
2. One request with 4,000 tokens of prompt plus output: 524,288 x 4,000 = 2.10 GB.
3. A batch of 16 such requests: 2.10 x 16 = 33.6 GB.
4. The FP16 weights are 14 GB. On a 48 GB GPU that leaves 34 GB, so 16 requests of this length fit. On an 80 GB GPU, 66 GB is left, so 31 fit.
5. Quantize the weights to 4-bit (3.5 GB) on a 24 GB GPU: 20.5 GB is left, so 9 requests fit. The cache, not the weights, now sets the limit.
6. With grouped-query attention and 8 KV heads, the cache is 4 times smaller: 0.52 GB per request.

Real servers also keep a few GB for activations, so treat these as upper bounds.

Latency for one request. Assume a 2,000-token prompt, 300 output tokens, 40 ms of queue wait, prefill at 10,000 tokens per second and a TPOT of 25 ms.

1. Prefill: 2,000 / 10,000 = 200 ms. TTFT = 40 + 200 = 240 ms.
2. Tokens per second: 1 / 0.025 = 40.
3. End to end: 0.240 + 299 x 0.025 = 7.715 s. Decode is 97% of it.
4. Halve the prompt: 7.615 s, a 0.1 s gain. Cap the output at 150 tokens: 0.240 + 149 x 0.025 = 3.965 s. Output length is the bigger lever.
5. Cache a 1,500-token prefix: prefill covers 500 tokens (50 ms), so TTFT falls from 240 ms to 90 ms.

Latency against throughput. Suppose 32 requests share the batch and TPOT rises to 40 ms. Each user now gets 25 tokens per second and the same request takes 0.240 + 299 x 0.040 = 12.2 s. The server produces 32 / 0.040 = 800 tokens per second, 20 times the single-request rate, for 1.6 times the per-token latency.

"""Static vs continuous batching on one GPU, plus a memory calculator.

Time is counted in decode steps: one step produces one token for every
request in the running batch. Prefill is ignored and a step costs the same
at any batch size, so the numbers show the scheduling effect only.
Pure Python, no NumPy.
"""
import random


def static_batching(reqs, slots, max_wait):
    """Fire a batch when `slots` requests are waiting or the oldest has waited
    `max_wait` steps. Hold the whole batch until its longest request finishes."""
    pending = sorted(reqs, key=lambda r: r[1])
    t, busy, waits, latencies = 0, 0, [], []
    while pending:
        t = max(t, pending[0][1])                       # GPU is free; first request is here
        fire_at = max(t, pending[0][1] + max_wait)      # latest moment we may fire
        if len(pending) >= slots:
            fire_at = min(fire_at, max(t, pending[slots - 1][1]))   # batch is full earlier
        t = fire_at
        batch = [r for r in pending if r[1] <= t][:slots]
        pending = [r for r in pending if r not in batch]
        longest = max(out for _, _, out in batch)
        for _, arrival, out in batch:
            waits.append(t - arrival)
            latencies.append(t + longest - arrival)     # everyone returns together
            busy += out                                 # useful slot-steps
        t += longest
    return report(waits, latencies, busy, slots, t)


def continuous_batching(reqs, slots):
    """At every decode step, drop finished requests and admit waiting ones."""
    pending = sorted(reqs, key=lambda r: r[1])
    running = {}                                        # name -> (arrival, tokens left)
    t, busy, waits, latencies = 0, 0, [], []
    while pending or running:
        while pending and pending[0][1] <= t and len(running) < slots:
            name, arrival, out = pending.pop(0)
            running[name] = (arrival, out)
            waits.append(t - arrival)
        if not running:                                 # nothing to do: jump to next arrival
            t = pending[0][1]
            continue
        busy += len(running)
        t += 1                                          # one decode step for the whole batch
        for name, (arrival, left) in list(running.items()):
            if left == 1:
                latencies.append(t - arrival)
                del running[name]                       # slot is free for the next step
            else:
                running[name] = (arrival, left - 1)
    return report(waits, latencies, busy, slots, t)


def report(waits, latencies, busy, slots, end):
    n = len(waits)
    return {"avg_wait": sum(waits) / n, "avg_latency": sum(latencies) / n,
            "end": end, "gpu_busy": busy / (slots * end)}


def show(title, reqs, slots, max_wait):
    print(title)
    for name, r in (("static", static_batching(reqs, slots, max_wait)),
                    ("continuous", continuous_batching(reqs, slots))):
        print(f"  {name:<10} avg wait {r['avg_wait']:7.2f}  avg latency {r['avg_latency']:7.2f}"
              f"  done at step {r['end']:4d}  GPU busy {r['gpu_busy']:.1%}")


def weight_gb(params_billion, bits):
    return params_billion * 1e9 * bits / 8 / 1e9


def kv_cache_bytes(layers, kv_heads, head_dim, seq_len, batch, bytes_per_value=2):
    return 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_value


if __name__ == "__main__":
    # The six requests from the picture: (name, arrival step, output tokens).
    six = [("A", 0, 2), ("B", 0, 8), ("C", 0, 4), ("D", 0, 3), ("E", 0, 5), ("F", 0, 2)]
    show("Six requests, 3 slots, all arrive at step 0", six, slots=3, max_wait=0)
    s, c = static_batching(six, 3, 0), continuous_batching(six, 3)
    assert (s["end"], c["end"]) == (13, 9)
    assert abs(s["avg_wait"] - 4.0) < 1e-9 and abs(c["avg_wait"] - 11 / 6) < 1e-9

    rng = random.Random(7)
    lengths = [20, 40, 80, 160, 400]
    burst = [(f"b{i}", 0, rng.choice(lengths)) for i in range(200)]
    show("Burst: 200 requests waiting at step 0, 8 slots", burst, slots=8, max_wait=0)

    t, steady = 0, []
    for i in range(200):
        t += rng.randint(10, 50)                        # a new request every 10 to 50 steps
        steady.append((f"s{i}", t, rng.choice(lengths)))
    load = sum(out for _, _, out in steady) / (8 * t)
    show(f"Steady: 200 requests, 8 slots, offered load {load:.0%} of the GPU", steady, slots=8, max_wait=20)

    print("Weights")
    for params, bits, label in ((7, 16, "7B FP16"), (7, 8, "7B INT8"), (7, 4, "7B 4-bit"), (70, 16, "70B FP16")):
        print(f"  {label:<9} {weight_gb(params, bits):6.1f} GB")

    print("KV cache: 32 layers, 32 heads, head dim 128, FP16")
    per_token = kv_cache_bytes(32, 32, 128, seq_len=1, batch=1)
    per_request = kv_cache_bytes(32, 32, 128, seq_len=4000, batch=1)
    print(f"  per token                {per_token:,} bytes ({per_token / 1e6:.2f} MB)")
    print(f"  one 4,000-token request  {per_request / 1e9:.2f} GB")
    print(f"  batch of 16              {kv_cache_bytes(32, 32, 128, 4000, 16) / 1e9:.1f} GB")
    for gpu_gb, w in ((24, weight_gb(7, 4)), (48, weight_gb(7, 16)), (80, weight_gb(7, 16))):
        fit = int((gpu_gb - w) * 1e9 // per_request)
        print(f"  {gpu_gb} GB GPU, {w:.1f} GB weights: {fit} requests of 4,000 tokens fit")
    gqa = kv_cache_bytes(32, 8, 128, 4000, 1)
    print(f"  with 8 KV heads (grouped-query attention): {gqa / 1e9:.2f} GB per request")