Most GenAI interviews open here. "What is a token?", "Explain self-attention simply", "Temperature vs top-p?" and "Why do LLMs hallucinate?" are warm-up questions, and a weak answer colours the rest of the round. The same ideas decide real engineering work: token counts set your bill and your latency, the context window sets how much history and retrieved text fits, and decoding settings decide whether an extraction job returns the same JSON twice. If you can draw the next-token loop and one transformer block from memory, the later topics (RAG, fine-tuning, serving) become details added to a picture you already have.

One objective: predict the next token

An LLM is a neural network trained to predict the next token. Chat, code and reasoning all come from that one objective, trained at very large scale.

At run time the model works in a loop. Text is split into tokens. The network outputs a probability for every token in its vocabulary. One token is chosen and appended to the input, and the loop runs again until a stop token appears. Every section below is one stage of that loop.

Run naively, each pass would recompute the whole sequence to produce one token. Real servers keep a KV cache: the attention keys and values of earlier tokens (defined in the self-attention section below) are stored, so each step computes only the new token.

Tokens and BPE

A token is the unit the model reads, usually a word piece. Roughly 1 token is 0.75 English words, so 1,000 tokens is about 750 words.

Why word pieces? A vocabulary of whole words can never cover every name, typo and new word. Single characters cover everything but make sequences very long. Word pieces sit between the two: a common word is one token, and a rare word splits into a few known pieces.

Tokenizers such as BPE (byte pair encoding) build a vocabulary by repeatedly merging the most frequent character pairs:

1. Start with every word split into single characters.
2. Count every adjacent pair of symbols across the corpus.
3. Merge the most frequent pair into one new symbol and add it to the vocabulary.
4. Repeat until the vocabulary reaches the size you want.

Here is a tiny corpus, as word counts: play 5, plays 3, played 4, player 2, day 7, plan 2.

Check merge 1: "ay" occurs in play, plays, played, player and day, so 5 + 3 + 4 + 2 + 7 = 21. After four merges the unseen word "replay" encodes as r, e, play: three tokens and no "unknown word" symbol. To encode new text you apply the merges in the order they were learned. Real tokenizers run tens of thousands of merges and usually start from bytes, so any string can be encoded.

Two practical effects follow. The 0.75 ratio is for English prose: code and text in other languages usually need more tokens per word, so measure with the real tokenizer. And the model sees pieces, not letters, which is why "how many r's are in strawberry" is harder for it than it looks.

The embedding table

A token ID is only an index, so the model needs numbers it can do maths on. The embedding table is a matrix with one row per vocabulary entry and d_model columns, where d_model is the length of every token vector inside the model. Token ID 4,821 selects row 4,821. That row is the token's vector, called its embedding.

The rows are learned during training. Tokens that appear in similar contexts end up with similar rows, so similar meanings sit close together in that vector space.

The table gives one fixed vector per token. "bank" gets the same row in "river bank" and "bank loan". It is the transformer blocks that mix in the neighbours and turn this fixed vector into one that fits the sentence.

The table is large. With a vocabulary of 50,000 and d_model = 4,096 it holds 50,000 x 4,096 = 204.8 million parameters. At the other end of the network, an output layer maps the final vector back to 50,000 scores called logits, one per vocabulary token.

Position

Attention has no sense of order: shuffle the input tokens and each token still gets the same result. "Dog bites man" and "man bites dog" would look identical, so position is added.

RoPE stands for rotary position embeddings. Because both the query and the key are rotated, their dot product depends on the difference of the two angles. The model learns "three tokens back", not "position 507".

Self-attention

Self-attention is the step where tokens exchange information. Every token looks at every other token and decides how much each one matters. It builds a query, a key and a value per token. The query is compared with every key, and the scores decide how much of each value to take.

In a generating model a causal mask hides later tokens, so a token attends only to itself and what came before. The next lesson builds this formula step by step with numbers.

The transformer block

A transformer block is multi-head attention, then a feed-forward layer, each wrapped in a residual connection and layer norm.

• Multi-head attention moves information between tokens. Several attention heads run in parallel.
• Feed-forward layer: two linear layers with a non-linearity between them, applied to each token on its own. It typically expands the vector to about 4 x d_model and brings it back.
• Residual connection: the input of a sub-layer is added to its output, x + f(x). Gradients get a direct path through the whole stack.
• Layer norm: rescales each token's vector to zero mean and unit variance, which keeps values in a stable range.

One pre-norm decoder block: a residual path with an attention branch and a feed-forward branch

This order is called pre-norm: normalise first, then run the sub-layer. The original transformer used post-norm, Norm(x + f(x)). Most modern LLMs use pre-norm because the residual path is never rescaled, and deep stacks train more stably. The input and output have the same shape, so you stack dozens of blocks.

Where do the parameters sit? Attention has four d_model x d_model matrices (query, key, value, output), so 4 x d_model^2. A feed-forward layer with 4 x expansion has two matrices of size d_model x 4 d_model, so 8 x d_model^2. With d_model = 4,096 that is about 67 million for attention and 134 million for the feed-forward layer: two thirds of a block's parameters are in the feed-forward part.

Three architectures

Most LLMs today are decoder-only.

Training stages

1. Pretraining on raw text (next-token prediction). The label is the next token of the text itself, so no human labelling is needed. The loss at each position is the negative log of the probability given to the correct token: 0.69 if the model gave it 0.5, and 4.6 if it gave it 0.01. With a causal mask, a 1,000-token text gives 999 predictions in one pass. The result is a base model that continues text.
2. Supervised fine-tuning on instructions. Instruction and response pairs teach the model to answer a request, not just continue it.
3. Preference alignment (RLHF or DPO). RLHF is reinforcement learning from human feedback. DPO is direct preference optimisation. Both use human choices between two answers to push the model toward the preferred one.

Reasoning models add reinforcement learning with verifiable rewards: the model is rewarded when a program can check the result, such as a correct final number or passing unit tests.

Decoding

The model outputs a probability for every token. Decoding is the rule that picks one.

Here z is the logit vector. A common order is: apply temperature, filter with top-k or top-p, renormalise so the kept probabilities sum to 1, then sample. As T approaches 0 the result approaches greedy. The worked example below does all of this with real numbers.

Context window

The context window is the maximum number of tokens the model can attend to in one call, input plus output.

Treat it as a budget. Assume an 8,000-token window. A 600-token system prompt, 2,400 tokens of chat history and 6 retrieved chunks of 500 tokens use 600 + 2,400 + 3,000 = 6,000 tokens. That leaves 2,000 tokens for the answer, about 1,500 words. If each new turn adds 800 tokens of history, two more turns leave only 400 tokens for the answer and a third does not fit, so you must trim or summarise. Nothing is remembered between calls: anything the model should know must be inside the window again on the next call.

A longer window is not free. The cost of attention is quadratic in sequence length: doubling the tokens gives four times as many token pairs to score.

Why hallucination happens

A hallucination is a fluent answer that is false. The model is trained to produce likely text, not verified facts: the training loss rewards a plausible next token, and no step checks truth. When knowledge is missing, outdated or ambiguous the model still produces a fluent answer, because every decoding step must output some token. "I don't know" appears only when it is the likely continuation.

Fixes: grounding with RAG (retrieval-augmented generation), asking for citations, lower temperature, letting the model say "I don't know", and evals that measure faithfulness.

Mixture of experts

In a Mixture of Experts (MoE) model the feed-forward layer is split into many expert networks, and a router sends each token to only a few of them. The model has many parameters but uses a small part per token, so quality rises without the same rise in compute. With 8 experts and a router that picks 2, each token runs a quarter of the feed-forward parameters. Many recent open-weight models use sparse MoE designs. The cost is memory: every expert must still be loaded.

LLM vs SLM

An SLM is a small language model. Pick a small model for narrow, high-volume tasks such as routing, classification and extraction, or when you need low latency, low cost or on-device privacy. Use a large model for open-ended reasoning. A common production design uses both: the small model handles or routes the easy requests and the large one takes the hard ones.

Decoding one token by hand. The prompt is "I would like a cup of". Pretend the vocabulary has only five tokens, and the model gives these probabilities at temperature 1:

Greedy picks "tea" every time.

Temperature. Dividing logits by T is the same as raising each probability to the power 1/T and renormalising.

• T = 0.5 means squaring: 0.25, 0.04, 0.0225, 0.01, 0.0025. The sum is 0.325. Divide each by 0.325.
• T = 2.0 means square roots: 0.7071, 0.4472, 0.3873, 0.3162, 0.2236. The sum is 2.0814. Divide each by 2.0814.

Top-k = 2 (at T = 1) keeps tea and coffee. Their sum is 0.70, so tea becomes 0.50 / 0.70 = 0.714 and coffee 0.20 / 0.70 = 0.286.

Top-p = 0.9 (at T = 1) adds probabilities from the top: 0.50, 0.70, 0.85, 0.95. The total first reaches 0.9 at the fourth token, so four tokens stay and "sugar" is dropped. Divide by 0.95: 0.526, 0.211, 0.158, 0.105.

What to notice. The odd answer "sugar" has probability 0.05 at T = 1. It falls to 0.008 at T = 0.5, rises to 0.107 at T = 2.0, and is exactly 0 under top-p = 0.9. Temperature reshapes every probability but never removes a token. Top-k and top-p remove the tail but keep the ratios between the survivors: tea is still 2.5 times as likely as coffee.

"""Temperature, top-k and top-p on the five-token distribution from the worked example."""
import numpy as np

TOKENS = ["tea", "coffee", "water", "milk", "sugar"]
PROBS = np.array([0.50, 0.20, 0.15, 0.10, 0.05])
LOGITS = np.log(PROBS)          # any logits whose softmax gives PROBS


def softmax(z):
    z = z - np.max(z)           # stable: the largest exponent is 0
    e = np.exp(z)
    return e / e.sum()


def with_temperature(logits, t):
    """Divide the logits by t, then softmax. t < 1 sharpens, t > 1 flattens."""
    return softmax(logits / t)


def top_k(probs, k):
    """Keep the k most likely tokens, zero the rest, renormalise."""
    keep = np.argsort(-probs)[:k]
    out = np.zeros_like(probs)
    out[keep] = probs[keep]
    return out / out.sum()


def top_p(probs, p):
    """Keep the smallest set of top tokens whose probability adds up to at least p."""
    order = np.argsort(-probs)
    cumulative = np.cumsum(probs[order])
    n_keep = int(np.searchsorted(cumulative, p)) + 1   # first index where the sum reaches p
    out = np.zeros_like(probs)
    out[order[:n_keep]] = probs[order[:n_keep]]
    return out / out.sum()


def show(label, probs):
    cells = "  ".join(f"{t}={x:.3f}" for t, x in zip(TOKENS, probs))
    print(f"{label:<12} {cells}")


if __name__ == "__main__":
    show("T = 1.0", with_temperature(LOGITS, 1.0))
    show("T = 0.5", with_temperature(LOGITS, 0.5))
    show("T = 2.0", with_temperature(LOGITS, 2.0))
    show("top-k = 2", top_k(PROBS, 2))
    show("top-p = 0.9", top_p(PROBS, 0.9))

    # The numbers from the worked example.
    assert np.allclose(with_temperature(LOGITS, 0.5), PROBS ** 2 / np.sum(PROBS ** 2))
    assert np.allclose(with_temperature(LOGITS, 2.0).round(3), [0.340, 0.215, 0.186, 0.152, 0.107])
    assert np.allclose(top_k(PROBS, 2).round(3), [0.714, 0.286, 0, 0, 0])
    assert np.allclose(top_p(PROBS, 0.9).round(3), [0.526, 0.211, 0.158, 0.105, 0])

    # Sample 10,000 next tokens at each temperature and count how often "sugar" appears.
    rng = np.random.default_rng(0)
    for t in (0.5, 1.0, 2.0):
        draws = rng.choice(len(TOKENS), size=10_000, p=with_temperature(LOGITS, t))
        print(f"T = {t}: 'sugar' sampled {np.mean(draws == 4):.3f} of the time")
    print("greedy pick:", TOKENS[int(np.argmax(PROBS))])