"Write an agent loop without a framework" is a common coding round, and "design a customer support agent" is a common design round. Both test the same thing: do you know what happens between the model's reply and the next model call? Interviewers ask what the model actually returns when it calls a tool, who runs the tool, what happens when the tool fails, and what stops a run that never finishes. Frameworks hide these answers. Once you have written the loop yourself in about 150 lines, you can read any framework, debug a stuck agent from its trace, and say which limits you would set and why.

An LLM in a loop

One LLM call is one question and one answer. An agent puts that call inside a loop and gives it tools, which are functions the model may ask your code to run. Six ideas cover the topic.

• Agent loop: read the goal, decide the next step, call a tool, observe the result, repeat. It needs a stop condition and a step limit.
• Workflow vs agent: in a workflow you code the path and the LLM fills in steps. In an agent the model chooses the path. Use the simplest one that solves the task.
• Tool calling: each tool has a name, a description and a JSON schema for its arguments. The model outputs a structured call, your code runs it, and the result goes back into the context.
• ReAct: the model alternates reasoning and acting, so each action is based on the latest observation.
• Planning: plan-and-execute makes a plan first and then runs the steps. Reflection has the model review its own output and retry.
• Human in the loop: pause for approval before actions that cost money or cannot be undone. This needs saved state so the run can resume.

Agent = model + harness

The model is stateless. It receives text and returns text. It cannot run code, it does not remember the previous call, and it cannot stop itself. The harness is everything around the model that makes it an agent.

Larger harnesses add three more parts, which the mini framework leaves out.

• Hooks: your own code that the harness runs at fixed points, such as before and after every tool call. A hook can log the call, block it or change it. The schema check in execute sits where a before-tool hook would run.
• Skills: saved instructions for one kind of task, such as "how to process a return". Only a one-line description stays in the context. The full text is loaded when a task needs it.
• Sub-agents: a second loop with its own message list. It does one sub-task and returns a short summary, so its tool results never enter the main list.

When two products use the same model and behave differently, the harness is the difference.

Step 1: the message list

The whole state of a run is one list. Each message has a role: system (rules), user (the task), assistant (what the model said) and tool (what a tool returned).

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": "What does order A123 cost in total with 18% GST?"},
]

Because the model remembers nothing, every call sends the whole list again. The list is the agent's short-term memory. It is also why each step costs more than the one before.

Step 2: a tool is a name, a description and a schema

"get_order": {
    "description": "Look up one order by its id. Use it before answering any question about an order.",
    "schema": {"properties": {"order_id": {"type": "string"}}, "required": ["order_id"]},
    "fn": get_order,
}

The model never sees fn. It sees the name, the description and the schema, and it chooses a tool by reading the description. So the description is part of the prompt: say when to use the tool, not only what it does. The schema is a contract that your code checks before anything runs.

Five rules make a good tool: one clear job, a description that says when to use it, typed arguments, short and relevant results, and errors that tell the model how to fix the call.

Step 3: the registry

The registry is a dictionary from tool name to tool. It does two jobs. It is rendered into text and sent to the model as the tool definitions. It is also the allow-list: a name that is not in the registry never runs, whatever the model writes.

Step 4: the loop

for step in range(1, max_steps + 1):
    reply = model(messages)                               # the model decides
    messages.append({"role": "assistant", "content": reply})
    decision = json.loads(reply)                          # parse the tool call
    if "final" in decision:
        return decision["final"]                          # stop condition
    observation = execute(decision["tool"], decision["args"])   # your code runs the tool
    messages.append({"role": "tool", "content": observation})   # feed the observation back
return "stopped: step limit"

Each pass does four things: call the model, parse its reply, run the tool, and append the result (the observation) so the next call can see it. In this framework the model replies with JSON text such as {"tool": "get_order", "args": {"order_id": "A123"}}. Provider APIs do the same thing with structured fields: you pass the tool definitions as a request parameter, the reply contains a tool name with JSON arguments, and you send the result back as a new message. In both cases the model does not run anything. Your application validates and executes the call.

Step 5: errors come back as readable text

Three things go wrong: the model names a tool that does not exist, the arguments fail the schema, or the function itself raises. None of these may crash the loop. execute catches all three and returns a string:

ERROR: missing required argument 'order_id'. get_order expects {"order_id": {"type": "string"}}

The model reads this as its next observation and usually fixes the call on the next step. A stack trace or a bare "400 Bad Request" gives it nothing to act on. A good error says what was wrong and what a correct call looks like.

Step 6: the exits

A loop with one exit is a bug waiting for a confused model. The harness owns every exit.

• Stop condition: the model returns a final answer and no tool call. This is the normal exit.
• Step limit: max_steps. If good runs need 6 steps, a cap of 12 is reasonable.
• Cost budget: count the tokens of every call and stop before the task spends more than its allowance. Production harnesses also cap wall-clock time and money.
• Repeated call: the same tool with the same arguments three times is a loop. Stop, or tell the model to change approach.
• Human approval: the run pauses before a risky tool and waits.

The budget check runs before each model call, so the run can never overspend:

prompt_tokens = sum(count_tokens(m["content"]) for m in messages)   # the whole list is sent again
if spent + prompt_tokens > budget_tokens:
    return "stopped: budget"

A stopped run should still report what it finished and why it stopped. "Stopped at the step limit after 12 steps, order found, refund not issued" is useful to the user and to whoever reads the trace.

The agent loop as a cycle of model decides, tool runs and observation returns, with four marked exits

One full trace

The task is "What does order A123 cost in total with 18% GST?". It needs two tools: get_order and calculator. The model's first call uses a wrong argument name, and the error text lets it recover.

A trace like this, saved for every run, is the main debugging tool for agents. The code example prints exactly this table.

From the mini framework to production

The loop stays the same. Five things change around it.

1. Real token counts. The API response reports the input and output tokens of each call. Use those numbers for the budget, not a characters-divided-by-4 estimate.
2. Several tool calls in one reply. A model may ask for two tools at once. Run each call and append one result per call, matched by the call's id.
3. Timeouts and retries on tools. Tools call networks, and networks fail. Give every tool call a timeout. Retry failures that may pass on a second try (timeouts, rate limits) with exponential backoff: wait 1 s, 2 s, 4 s, each with a little random jitter, then give up and return an error text. Retry a write only if repeating it is safe.
4. A saved trace. Store every step with its model output, tool result, tokens and latency. You cannot debug or evaluate an agent you cannot replay.
5. An eval set. Measure the outcome (task success rate) and the path (right tools, right order, number of steps, cost, latency) on a fixed set of tasks, and add every failure you find as a new task.

ReAct, planning and reflection

The loop above becomes ReAct when the model writes a short thought before each call: thought, action, observation, repeat. Each action uses the newest observation, so the agent can change direction after a surprise. Plan-and-execute spends the first call on a numbered plan, runs the steps one by one, and replans when a step fails. It suits long tasks, but a plan goes stale when the facts change. Reflection adds a review call: the model checks its own draft against the criteria and retries. It costs extra calls, so cap the number of retries.

Workflow or agent

If you can write the steps down in advance, build a workflow: classify the ticket, retrieve the policy, draft the reply. You code the path and the LLM fills in each step. Use an agent only when the path depends on what the model finds along the way, such as debugging a failing test. Workflows are cheaper, faster and easier to test, so the agent has to earn its place. Do not build an agent when a single prompt or a fixed workflow solves the task, when mistakes are expensive and cannot be reviewed, when latency must be very low, or when you have no way to evaluate the result.

Human in the loop

Mark risky tools, such as issue_refund, as needing approval. When the model asks for one, the harness does not run it. It saves the message list and the pending call (a checkpoint) and returns "waiting for approval". Hours later a person answers. On yes, the harness loads the checkpoint, runs the tool and continues the loop. On no, it appends "ERROR: a person rejected this action" as the observation and the model picks another path. Without saved state the process would have to stay alive for hours, or the task would start again from zero.

What goes wrong, and what the harness does about it

Long tasks are the hardest, because small mistakes add up.

What a 10-step run costs. Assume the system prompt and tool definitions take 1,500 tokens and the task takes 100. On each step the model writes 150 tokens (a short thought and a tool call) and the tool returns 600 tokens. Both join the message list, so the list grows by 750 tokens per step. Every call sends the whole list again:

Input tokens at step n = 1,600 + 750 x (n - 1)

Total input over 10 steps = 10 x 1,600 + 750 x (0 + 1 + ... + 9) = 16,000 + 750 x 45 = 49,750 tokens. Total output = 10 x 150 = 1,500 tokens.

At an assumed 3 USD per million input tokens and 15 USD per million output tokens: 49,750 x 3 / 1,000,000 = 0.149 USD, plus 1,500 x 15 / 1,000,000 = 0.0225 USD. About 0.17 USD per task.

Double the steps. For 20 steps the sum is 20 x 1,600 + 750 x 190 = 174,500 input tokens. Twice the steps cost 3.5 times the tokens. This is why a step limit is also a cost limit.

Trim the tool results. If each tool returns 200 tokens, not 600, the list grows by 350 per step. Ten steps cost 16,000 + 350 x 45 = 31,750 input tokens, 36% fewer.

Why small errors add up. Suppose each step is right 95% of the time and one wrong step ruins the task. A 10-step task succeeds 0.95^10 = 60% of the time, and a 20-step task 0.95^20 = 36%. At 99% per step, 10 steps succeed 0.99^10 = 90% of the time. Schema checks and readable errors let the model repair a bad step, so it does not carry the mistake forward.

"""A mini agent framework: message list, tools, registry, loop, limits, trace.

The "model" is a scripted function, so this runs with no network and no API key.
Replace fake_model with a real chat API call and the harness stays the same.
"""
import ast
import json
import operator
from collections import Counter

# ---- Tools: plain functions -------------------------------------------------
ORDERS = {"A123": {"item": "keyboard", "unit_price": 2500, "qty": 2}}
OPS = {ast.Add: operator.add, ast.Sub: operator.sub, ast.Mult: operator.mul, ast.Div: operator.truediv}


def get_order(order_id):
    if order_id not in ORDERS:
        raise ValueError(f"no order with id {order_id}; ids look like A123")
    return ORDERS[order_id]


def calculator(expression):
    def ev(node):                                   # walk the syntax tree, never eval()
        if isinstance(node, ast.Constant) and isinstance(node.value, (int, float)):
            return node.value
        if isinstance(node, ast.BinOp) and type(node.op) in OPS:
            return OPS[type(node.op)](ev(node.left), ev(node.right))
        raise ValueError("only numbers and + - * / are allowed")
    return round(ev(ast.parse(expression, mode="eval").body), 2)


# ---- Registry: name -> description + argument schema + function ---------------
REGISTRY = {
    "get_order": {
        "description": "Look up one order by its id. Use it before answering any question about an order.",
        "schema": {"properties": {"order_id": {"type": "string"}}, "required": ["order_id"]},
        "fn": get_order,
    },
    "calculator": {
        "description": "Evaluate arithmetic such as '2500 * 2 * 1.18'. Use it for every calculation.",
        "schema": {"properties": {"expression": {"type": "string"}}, "required": ["expression"]},
        "fn": calculator,
    },
}
TYPES = {"string": str, "number": (int, float)}
SYSTEM = ('Reply with JSON only: {"tool": name, "args": {...}} to act, or {"final": text} to finish.\nTools:\n'
          + "\n".join(f"- {n}: {t['description']} Arguments: {json.dumps(t['schema'])}" for n, t in REGISTRY.items()))


def execute(name, args):
    """Run one tool call. Never raises: every failure becomes text the model can read."""
    tool = REGISTRY.get(name)
    if tool is None:
        return f"ERROR: unknown tool '{name}'. Available tools: {sorted(REGISTRY)}"
    props = tool["schema"]["properties"]
    hint = f"{name} expects {json.dumps(props)}"
    if not isinstance(args, dict):
        return f"ERROR: args must be a JSON object. {hint}"
    for key in tool["schema"]["required"]:
        if key not in args:
            return f"ERROR: missing required argument '{key}'. {hint}"
    for key, value in args.items():
        if key not in props or not isinstance(value, TYPES[props[key]["type"]]):
            return f"ERROR: bad argument '{key}'. {hint}"
    try:
        return json.dumps(tool["fn"](**args))
    except Exception as exc:                        # a tool crash must not crash the loop
        return f"ERROR: {name} failed: {exc}"


def count_tokens(text):
    return max(1, len(text) // 4)                   # rough rule: about 4 characters per token


def run_agent(task, model, max_steps=6, budget_tokens=3000):
    messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": task}]
    spent, seen = 0, Counter()
    for step in range(1, max_steps + 1):
        prompt_tokens = sum(count_tokens(m["content"]) for m in messages)   # the whole list is sent again
        if spent + prompt_tokens > budget_tokens:
            return "stopped: budget", None, spent
        reply = model(messages)                                             # 1. the model decides
        spent += prompt_tokens + count_tokens(reply)
        messages.append({"role": "assistant", "content": reply})
        try:                                                                # 2. parse the decision
            decision = json.loads(reply)
        except json.JSONDecodeError:
            decision = {}
        if "final" in decision:                                             # 3. stop condition
            print(f"  step {step} | final: {decision['final']} | tokens so far {spent}")
            return "done", decision["final"], spent
        seen[reply] += 1
        if seen[reply] == 3:
            print(f"  step {step} | {reply} | not run: third identical call")
            return "stopped: same call 3 times", None, spent
        if "tool" in decision:                                              # 4. run the tool
            observation = execute(decision["tool"], decision.get("args"))
        else:
            observation = 'ERROR: reply must be JSON with "tool" and "args", or with "final".'
        messages.append({"role": "tool", "name": decision.get("tool"), "content": observation})  # 5. feed it back
        print(f"  step {step} | {reply} | {observation} | tokens so far {spent}")
    return "stopped: step limit", None, spent


# ---- Scripted stand-ins for the LLM -------------------------------------------
def call(tool, **args):
    return json.dumps({"tool": tool, "args": args})


def fake_model(messages):
    last = messages[-1]
    if last["role"] == "user":
        return call("get_order", id="A123")                 # mistake: wrong argument name
    if last["content"].startswith("ERROR"):
        return call("get_order", order_id="A123")           # reads the error text and fixes the call
    if last["name"] == "get_order":
        order = json.loads(last["content"])
        return call("calculator", expression=f"{order['unit_price']} * {order['qty']} * 1.18")
    return json.dumps({"final": f"Order A123 costs {last['content']} rupees with 18% GST."})


def stuck_model(messages):
    return call("get_order", order_id="Z999")               # never changes its approach


def wandering_model(messages):
    return call("get_order", order_id=f"Z{len(messages)}")  # a new wrong guess every step


if __name__ == "__main__":
    task = "What does order A123 cost in total with 18% GST?"
    runs = [("recovers from a bad call", fake_model, 6, 3000), ("repeats itself", stuck_model, 6, 3000),
            ("wanders", wandering_model, 4, 3000), ("tight budget", fake_model, 6, 400)]
    results = {}
    for label, model, max_steps, budget in runs:
        print(f"{label} (max_steps={max_steps}, budget_tokens={budget})")
        results[label] = run_agent(task, model, max_steps, budget)
        print(f"  -> {results[label][0]}, {results[label][2]} tokens spent\n")
    assert results["recovers from a bad call"][:2] == ("done", "Order A123 costs 5900.0 rupees with 18% GST.")
    assert results["repeats itself"][0] == "stopped: same call 3 times"
    assert results["wanders"][0] == "stopped: step limit"
    assert results["tight budget"][0] == "stopped: budget"
    print("all four exits checked")