Lesson 9 explains what LoRA is. Interviewers then ask what you did with it: where the data came from, how you split it, how you knew the model was better, and what you checked when it was not. These are project questions, and "I trained it and it looked good" fails them. In real work most fine-tuning failures have nothing to do with rank or alpha. They come from duplicated or mislabelled rows, a chat template that differs between training and serving, a test set that leaked, or no baseline to compare with. This lesson gives you one complete project story with numbers that you can adapt to your own.

The use case

An online electronics store receives about 3,000 support tickets a day. For each ticket the system must return one JSON object: a category (one of 8), a priority (low, normal or urgent) and a reply draft of at most 80 words in the store's tone. A human agent checks the draft and sends it.

The whole lesson follows this case. Lesson 9 is the prerequisite: it explains LoRA, QLoRA, rank and alpha. This lesson covers the work around them.

Six fine-tuning stages in a column, from the decision to serving, with a gate between each pair and the action to take when a gate fails

Fine-tune, prompt or RAG?

Use the cheapest tool that passes the eval.

Fine-tuning changes behaviour, not knowledge. Do not train the refund policy into the weights: when the window changes from 7 to 10 days you would have to retrain. Keep facts in retrieval, and put the retrieved policy snippet into every training prompt, so the model practises using context the way it will in production.

Fine-tune only when you have a stable task, a real eval and roughly 500 or more good examples, and only after you have measured a prompt baseline. Here the best few-shot prompt reaches 80% category accuracy with a 2,600-token prompt, and the target is 90%. At 3,000 tickets a day that prompt costs 7.8 million input tokens a day. A tuned model with a 400-token prompt needs 1.2 million.

Building and cleaning the dataset

Each training example is one conversation in the messages format: a system message, a user message (ticket plus policy snippet) and an assistant message (the target JSON). The best source is your own resolved tickets, because they look like production inputs.

1. Filter for quality. Keep tickets that a human agent resolved without a reopen. A model trained on bad replies writes bad replies.
2. Mask PII (personal data). Replace emails, phone numbers and addresses with placeholders before the data leaves the ticket system.
3. Remove duplicates. Normalise case and whitespace, then drop repeats. Pasted macros can fill a large part of a support export, and a duplicate that lands in both train and test inflates the score.
4. Check the labels. Have two people relabel a sample of 200. If they often disagree with the stored label, fix the label guide first. The model cannot be more consistent than its labels.
5. Validate the targets. Every assistant message must parse as JSON and use only allowed labels. Malformed targets teach malformed output.
6. Check the length. Count tokens per example. The trainer cuts at max_length and keeps the start, so in a long example the part that is lost is the answer.
7. Balance. Cap the largest categories so "where is my order" does not fill most of the set, and keep every example of a rare class that matters, such as fraud.

How much data? For tone or instruction following, 500 to 2,000 clean examples often beat tens of thousands of noisy ones. Domain adaptation needs more, around 5,000 to 20,000. Preference tuning with DPO needs about 1,000 to 5,000 pairs. If real examples are too few, a larger model can draft targets for real inputs. People must still review a sample, and the test set stays human-labelled.

Chat template and special tokens

Lesson 6 explains templates. For training, five rules matter.

• Use the base model's own template. The trainer applies it to a messages column. Print one rendered example and read it before any run.
• End every assistant turn with the end-of-turn token. That token is how the model learns to stop. A tuned model that keeps writing after the JSON usually never saw it in training.
• Keep padding away from the end token. If padding reuses the end token and your collator (the function that pads examples into a batch) removes every position with that id from the loss, the end token is never trained.
• Train on the answer. By default the loss covers every token, including the ticket text. assistant_only_loss=True in SFTConfig limits it to assistant tokens, if the template supports it.
• Serve what you trained. Same system prompt, same message order, same template, plus add_generation_prompt=True at inference.

Avoid adding new special tokens. If you must, the embedding table has to be resized, trained and saved with the adapter.

Train, validation and test split

Hold out 10 to 20% of the data. The usual split is 80 / 10 / 10, and each part has one job.

• Train: the adapter learns from it.
• Validation: you choose the checkpoint and the hyperparameters on it.
• Test: you touch it once, to compare with the base model.

Split by customer, not by row. Two tickets from one customer share names, products and phrasing, so a random row split leaks. Remove duplicates before splitting, and check that every category appears in every split. If you can, take the test set from the most recent weeks, because production data is always newer than training data. Then freeze the test set and version it.

QLoRA hyperparameters

PEFT's own defaults are r = 8 and alpha = 8, so set both yourself. Lesson 9 starts LoRA on the attention projections; with a 4-bit base the usual start is all linear layers, following the QLoRA paper's finding. Gradient accumulation adds up the gradients of 4 small batches before one optimiser step, so you get the effect of a batch of 16 in the memory of a batch of 4. Warmup raises the learning rate from zero over the first steps, so early updates stay small. With 1,920 training examples and an effective batch of 16, one epoch is 120 optimiser steps and the full run is 360.

# needs: transformers, peft, trl, datasets, bitsandbytes, torch
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoTokenizer, BitsAndBytesConfig, EarlyStoppingCallback
from trl import SFTConfig, SFTTrainer

base_id = "HuggingFaceTB/SmolLM2-1.7B-Instruct"   # any instruct model your GPU can hold
data = load_dataset("json", data_files={"train": "train.jsonl", "validation": "validation.jsonl"})
tokenizer = AutoTokenizer.from_pretrained(base_id)
print(tokenizer.apply_chat_template(data["train"][0]["messages"], tokenize=False))  # read it

trainer = SFTTrainer(
    model=base_id,
    train_dataset=data["train"],
    eval_dataset=data["validation"],
    processing_class=tokenizer,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16),
    peft_config=LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                           target_modules="all-linear", task_type="CAUSAL_LM"),
    args=SFTConfig(
        output_dir="ticket-triage-adapter", max_length=1024,
        learning_rate=2e-4, warmup_steps=20, num_train_epochs=3,
        per_device_train_batch_size=4, gradient_accumulation_steps=4,
        logging_steps=10, eval_strategy="steps", eval_steps=30,
        save_strategy="steps", save_steps=30, load_best_model_at_end=True,
        metric_for_best_model="eval_loss", greater_is_better=False),
    callbacks=[EarlyStoppingCallback(early_stopping_patience=3)],
)
trainer.train()
trainer.save_model("ticket-triage-adapter/best")

EarlyStoppingCallback ends the run after 3 evaluations without a lower validation loss, and load_best_model_at_end reloads the best checkpoint before saving.

The script uses a 1.7-billion-parameter model so it runs on a free notebook GPU. The 4-bit load is what lets you swap in an 8-billion-parameter model on the same 16 GB card: 16 GB of weights in 16-bit, about 4 GB in 4-bit.

Reading loss curves

Training loss is logged every 10 steps and validation loss every 30. Read the two together.

A model that guesses evenly over a 32,000-token vocabulary has a loss of ln 32,000 = 10.4. A pretrained model on well-formatted text starts far below that. Loss also is not the goal: it picks the checkpoint, and task metrics decide whether you ship.

Evaluating against the base model

Score the base and the fine-tuned model on the same test prompts with task metrics and an LLM judge, then run a regression check on general tasks. If the gain is not clear, do not ship.

• Three contenders: the base model with its best zero-shot prompt, the base model with the few-shot prompt (the real alternative), and the tuned model. Same test tickets, greedy decoding.
• One metric per field: JSON validity by a code check, category accuracy and recall on urgent tickets by exact match, reply quality by a judge model with a rubric that you have checked against about 50 human ratings.
• Know the noise. With 240 test tickets and 80% accuracy, the standard error is sqrt(0.8 x 0.2 / 240) = 2.6 points, so the 95% range is about plus or minus 5 points. A 2-point gain proves nothing.
• Check for catastrophic forgetting: the model gets better at the new task and worse at things it knew before. Run about 100 general prompts through both models.
• Read 20 raw outputs. Metrics miss a reply that is valid JSON and rude.

Merging and serving the adapter

# needs: transformers, peft, torch (base_id and tokenizer come from the training script)
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.bfloat16)   # 16-bit, not 4-bit
merged = PeftModel.from_pretrained(base, "ticket-triage-adapter/best").merge_and_unload()
merged.save_pretrained("ticket-triage-merged")
tokenizer.save_pretrained("ticket-triage-merged")

# needs: vllm
vllm serve HuggingFaceTB/SmolLM2-1.7B-Instruct --enable-lora \
    --lora-modules ticket-triage=ticket-triage-adapter/best

With vLLM a request selects the adapter by putting ticket-triage in the model field. Merge into the 16-bit base, then run the test set again on the exact artefact you serve, because the adapter was trained against 4-bit weights. Record the base model revision, the dataset version and the system prompt with the adapter. Roll out in shadow mode (the model runs on real tickets but agents do not see its output), then to a small share of traffic, called a canary. Rolling back means removing the adapter.

Debugging a model that got worse

Check in this order, cheapest first.

1. Template and special tokens. Print a rendered training example and the inference prompt and compare them character by character. This is the most frequent bug.
2. Is the adapter loaded? If outputs match the base model exactly, it is not.
3. Learning rate too high. The loss curve is spiky.
4. Too many epochs on small data. Validation loss rose and you kept the last checkpoint.
5. Noisy or mislabelled examples. Read 50 random training rows.
6. Training data that does not match production inputs. For example, you trained on cleaned tickets and production tickets carry signatures and forwarded threads.
7. The eval itself. Sampling left on, or max_new_tokens so small that the JSON is cut off.

The ticket-triage run from baseline to decision. The figures are for the example store. Follow the arithmetic, not the absolute values.

1. Baseline. On 240 test tickets, the base model with its best zero-shot prompt returns valid JSON for 214 (89.2%) and the right category for 170 (70.8%). With 8 few-shot examples in a 2,600-token prompt: 233 valid (97.1%) and 192 right (80.0%). The target is 90%.
2. Data funnel. 60,000 sampled resolved tickets. Quality filter (human-resolved, not reopened): 22,000 left. Duplicates removed: 9,500. Invalid targets and over-length examples removed: 9,100. Cap of 320 per category, with all 160 fraud tickets kept: 7 x 320 + 160 = 2,400.
3. Split by customer, 80 / 10 / 10. 1,920 train, 240 validation, 240 test. A real customer split lands a few rows off these round numbers.
4. Steps. Effective batch 16, so 1,920 / 16 = 120 steps per epoch and 360 for 3 epochs. Evaluating every 30 steps gives up to 12 validation points.
5. Size of the job. At an average of 420 tokens per example, one epoch is 1,920 x 420 = 806,400 tokens and the full run is 2,419,200. At an assumed 2,000 training tokens per second that is 1,210 seconds, about 20 minutes.
6. Loss curve. Validation loss is 1.12 at step 30, 0.78 at step 120 and 0.71 at step 240. The next three evaluations are all higher, ending at 0.74 at step 330, while training loss keeps falling to 0.36. With a patience of 3, early stopping ends the run there and reloads the step-240 checkpoint.
7. Test. The tuned model returns valid JSON for 240 of 240 and the right category for 218 (90.8%). That is 10.8 points above the few-shot prompt, against a noise range of about 5 points. It finds 31 of 36 urgent tickets (86%); the few-shot prompt found 24 (67%).
8. Regression. On 100 general prompts the base passes 93 and the tuned model 91. For 100 prompts at 93% the noise range is also about 5 points, so there is no sign of forgetting.
9. Decision. Ship behind a canary. The prompt shrinks from 2,600 to 400 tokens, so daily input drops from 3,000 x 2,600 = 7.8 million tokens to 3,000 x 400 = 1.2 million.

Had step 7 shown 82% against 80%, the right call would be not to ship: 2 points is inside the noise, and the prompt-only system has nothing to retrain.

"""Two parts of a fine-tuning pipeline that run without a GPU.

Part 1, the data gates: validate, mask PII, dedupe and split support tickets
        by customer, so no customer appears in two splits.
Part 2, a toy LoRA run in NumPy: a frozen "base" classifier gets a rank-4
        adapter. Read the two loss curves, keep the best checkpoint, and
        compare base, best and last on a held-out test set.
"""
import hashlib
import json
import math
import random
import re
from collections import Counter

import numpy as np

CATEGORIES = ["billing", "delivery", "returns", "account"]
PRIORITIES = ["low", "normal", "urgent"]
MAX_TOKENS = 120                                   # budget for one ticket
TEXTS = {
    "billing": "I was charged twice for order {o}. Please refund one payment.",
    "delivery": "Order {o} was due on Monday and has still not arrived.",
    "returns": "The headphones from order {o} are faulty. How do I send them back?",
    "account": "I cannot log in after changing my phone. Order {o} is stuck in my cart.",
}


def make_raw(n=400, seed=7):
    """Synthetic export of resolved tickets, with the defects real exports have."""
    rnd, rows = random.Random(seed), []
    for i in range(n):
        cat = rnd.choice(CATEGORIES)
        ticket = TEXTS[cat].format(o=10000 + rnd.randrange(90000))
        target = json.dumps({"category": cat, "priority": rnd.choice(PRIORITIES),
                             "reply": "Sorry about this. We are checking it now."})
        if i % 9 == 4:                                 # the same ticket filed again, in capitals
            ticket, target = "  " + rows[i - 2]["ticket"].upper(), rows[i - 2]["target"]
        if i % 11 == 0:
            ticket += " Mail me at user%d@example.com." % i   # PII to mask
        if i % 25 == 3:
            target = target[:-9]                       # cut off: not valid JSON
        if i % 31 == 5:
            target = target.replace(cat, "misc")       # label outside the schema
        if i % 40 == 7:
            ticket = ticket + " Please help." * 80     # far over the token budget
        rows.append({"customer": "c%03d" % rnd.randrange(90), "ticket": ticket, "target": target})
    return rows


def est_tokens(text):
    return math.ceil(len(text.split()) / 0.75)         # 1 token is about 0.75 words


def reject_reason(row):
    try:
        out = json.loads(row["target"])
    except json.JSONDecodeError:
        return "target is not valid JSON"
    if out.get("category") not in CATEGORIES or out.get("priority") not in PRIORITIES:
        return "label outside the schema"
    if not out.get("reply", "").strip():
        return "empty reply"
    if est_tokens(row["ticket"]) > MAX_TOKENS:
        return "ticket over the token budget"
    return None


def clean(rows):
    kept, seen, dropped = [], set(), Counter()
    for row in rows:
        reason = reject_reason(row)
        if reason:
            dropped[reason] += 1
            continue
        row = dict(row, ticket=re.sub(r"\S+@\S+\.\w+", "<EMAIL>", row["ticket"]).strip())
        key = hashlib.sha1(" ".join(row["ticket"].lower().split()).encode()).hexdigest()
        if key in seen:                                # same text after normalising
            dropped["duplicate"] += 1
            continue
        seen.add(key)
        kept.append(row)
    return kept, dropped


def split_by_customer(rows, seed=0):
    """80/10/10 over customers, not rows: one customer's tickets stay together."""
    customers = sorted({r["customer"] for r in rows})
    random.Random(seed).shuffle(customers)
    a, b = int(0.8 * len(customers)), int(0.9 * len(customers))
    part = {c: "train" for c in customers[:a]}
    part.update({c: "validation" for c in customers[a:b]})
    part.update({c: "test" for c in customers[b:]})
    splits = {"train": [], "validation": [], "test": []}
    for r in rows:
        splits[part[r["customer"]]].append(r)
    return splits


def softmax(z):
    z = z - z.max(axis=1, keepdims=True)
    e = np.exp(z)
    return e / e.sum(axis=1, keepdims=True)


def toy_run(lr, steps=400, seed=0):
    """Frozen W0 plus a trainable rank-4 update B A, trained on 120 noisy examples."""
    rng = np.random.default_rng(seed)
    d, k, r, scale = 4, 24, 4, 2.0                     # classes (outputs), features (inputs), rank, alpha / r
    W0 = rng.normal(0, 0.5, (d, k))                    # the frozen base, d x k as in lesson 9
    W_task = W0 + rng.normal(0, 0.8, (d, 2)) @ rng.normal(0, 0.8, (2, k))   # what the task needs

    def data(n, noise):
        X = rng.normal(size=(n, k))
        y = (X @ W_task.T).argmax(axis=1)
        flip = rng.random(n) < noise                   # mislabelled rows
        y[flip] = rng.integers(0, d, flip.sum())
        return X, y

    (Xtr, ytr), (Xva, yva), (Xte, yte) = data(120, 0.2), data(400, 0.2), data(2000, 0.0)
    A, B = rng.normal(0, 0.1, (r, k)), np.zeros((d, r))    # A is r x k, B is d x r and starts at 0

    def loss_acc(X, y, A, B):
        p = softmax(X @ (W0 + scale * B @ A).T)
        return float(-np.log(p[np.arange(len(y)), y] + 1e-12).mean()), float((p.argmax(1) == y).mean())

    curve, best = [], (np.inf, 0, A.copy(), B.copy())
    for step in range(steps + 1):
        tr, va = loss_acc(Xtr, ytr, A, B)[0], loss_acc(Xva, yva, A, B)[0]
        if step % 50 == 0:
            curve.append((step, tr, va))
        if va < best[0]:
            best = (va, step, A.copy(), B.copy())      # checkpoint with the lowest validation loss
        p = softmax(Xtr @ (W0 + scale * B @ A).T)
        p[np.arange(len(ytr)), ytr] -= 1               # gradient at the logits: p - y
        G = p.T @ Xtr / len(ytr)                       # gradient for the full matrix
        gB, gA = scale * G @ A.T, scale * B.T @ G      # chain rule into B and A; W0 gets nothing
        B, A = B - lr * gB, A - lr * gA
    acc = {"base": loss_acc(Xte, yte, A * 0, B * 0)[1],
           "best": loss_acc(Xte, yte, best[2], best[3])[1],
           "last": loss_acc(Xte, yte, A, B)[1]}
    return curve, best[1], acc


if __name__ == "__main__":
    print("PART 1. Data gates")
    raw = make_raw()
    kept, dropped = clean(raw)
    print(f"  raw rows: {len(raw)}   kept: {len(kept)}   dropped: {dict(dropped)}")
    assert not any("@" in r["ticket"] for r in kept), "PII left in the data"
    splits = split_by_customer(kept)
    owners = {name: {r["customer"] for r in rows} for name, rows in splits.items()}
    assert not (owners["train"] & owners["validation"] or owners["train"] & owners["test"]
                or owners["validation"] & owners["test"])
    for name, rows in splits.items():
        mix = Counter(json.loads(r["target"])["category"] for r in rows)
        print(f"  {name:10s} {len(rows):3d} rows, {len(owners[name]):2d} customers, {dict(sorted(mix.items()))}")
    print("  no customer appears in two splits")

    print("\nPART 2. Reading loss curves on a toy adapter")
    for lr, label in ((0.05, "sensible learning rate"), (1.0, "learning rate 20 times too high")):
        curve, best_step, acc = toy_run(lr)
        print(f"  {label} (lr = {lr})")
        for step, tr, va in curve:
            print(f"    step {step:3d}   train loss {tr:7.3f}   validation loss {va:7.3f}")
        print(f"    lowest validation loss at step {best_step}")
        print(f"    test accuracy: base {acc['base']:.3f}, best checkpoint {acc['best']:.3f}, "
              f"last checkpoint {acc['last']:.3f}")