The Agent Loop: How LLMs Become Actors

A startup ran their customer support agent in production for three days before the pager went off. The agent had been looping on a single order lookup — the inventory API was returning a 429, the agent kept retrying, and nobody had set an iteration cap. The bill for those three days was $4,200. The task itself, if it had succeeded, would have cost about $0.20.

This is the failure mode that the agent loop tutorial never shows you.

What the loop actually is

An agent is any system where an LLM output drives the choice of the next action, and that action's result feeds back as input to the next LLM call. That's it. No exotic architecture required. The minimal implementation is about 20 lines of Python.

The loop has five conceptual stages that every framework eventually reduces to:

Perceive — the model reads its full current context: system prompt, tool definitions, conversation history, prior tool results. Everything it knows about the task lives here. There is no other memory by default.

Reason — the model generates a response. For a capable model on a hard problem, this includes internal reasoning (visible as thinking tokens in models like Claude's extended thinking mode) before it commits to an action.

Plan — the model decides what to do next. In practice this is embedded in the reasoning step: the output is either a tool call or a final answer, and that choice is the plan.

Act — the model emits its decision. If it's a tool call, the output is a structured JSON object. If the task is done, the output is a text response. The model itself does nothing further: the host application is responsible for executing any tool calls.

Observe — the host application runs the tool, gets a result, and appends both the tool call and its result to the conversation. The updated context becomes the input to the next Perceive step.

sequenceDiagram
    participant U as User / Orchestrator
    participant M as LLM
    participant H as Host App
    participant T as Tool (API / DB / Code)

    U->>H: task + initial context
    loop Agent iteration
        H->>M: messages array (system + history)
        M-->>H: response (tool_use OR end_turn)
        alt tool_use
            H->>T: execute tool call
            T-->>H: tool result
            H->>H: append tool_use + tool_result to messages
        else end_turn
            H-->>U: final answer
        end
    end

The model never touches T directly. It never runs a network request. It emits JSON; your code runs the network request. This separation is not incidental — it is what makes agent behavior auditable and what lets you validate or reject tool calls before they execute.

The tool call in detail

When the model decides to call a tool, it emits a structured block. On Anthropic the stop reason is tool_use; on OpenAI it's tool_calls in the response. The payload looks like this:

# What the model returns (Anthropic SDK)
{
    "type": "tool_use",
    "id": "toolu_01XNrGbp...",
    "name": "get_order_status",
    "input": {
        "order_id": "ORD-4892",
        "include_shipping": True
    }
}

Your host application sees the stop reason, extracts this block, runs get_order_status(order_id="ORD-4892", include_shipping=True), and appends both the call and the result back to the conversation:

# Minimal agent loop (Anthropic SDK)
import anthropic

client = anthropic.Anthropic()

def run_agent(task: str, tools: list, tool_executor: callable,
              max_iterations: int = 10) -> str:
    messages = [{"role": "user", "content": task}]

    for iteration in range(max_iterations):
        response = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=4096,
            tools=tools,
            messages=messages,
        )

        # Append assistant response
        messages.append({"role": "assistant", "content": response.content})

        if response.stop_reason == "end_turn":
            # Extract text content from the final response
            for block in response.content:
                if block.type == "text":
                    return block.text
            return ""

        if response.stop_reason == "tool_use":
            tool_results = []
            for block in response.content:
                if block.type == "tool_use":
                    result = tool_executor(block.name, block.input)
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": str(result),
                    })
            messages.append({"role": "user", "content": tool_results})

    return "Max iterations reached without completing task."

This is the complete loop. Everything else — planning strategies, memory, multi-agent coordination — is built on top of this basic mechanism. The ReAct, Reflexion, and Planning Patterns article covers the prompting strategies that change how the model reasons inside this loop; the loop mechanics are the same.

Termination: the part that actually matters

The most common production mistake is treating stop_reason == "end_turn" as authoritative. It is not. The model believes it is done. It might be wrong. It might have hallucinated a success without actually completing the task. It might have gotten a cached tool result and assumed the task was complete.

You need three independent guards, enforced by your application, not the model:

1. Hard iteration cap. Set it before deployment. For most tasks 8-15 iterations is enough; anything requiring more is probably a sign the task is poorly scoped or the tools are inadequate. Cap it at 20 for complex workflows and treat any run that hits the cap as a failure to investigate, not a success.

2. Token budget. Check the remaining token budget before each LLM call. A loop that has consumed 80% of a 128k context window is increasingly unreliable — the model is attending to a degraded, overlapping history. Stop early and surface the partial result rather than letting the context rot.

Back-of-envelope: token cost per agent run

System prompt + tool definitions:  ~2,000 tokens (static, cacheable)
Per-iteration overhead (avg):       ~800 tokens (tool call + result)
Each API call re-bills the full growing prefix, so 10 iterations bill:
  (2,000 + 800) + (2,000 + 1,600) + ... + (2,000 + 8,000)
  = ~56,000 input tokens total, not 10,000
At claude-sonnet-4-5 pricing (mid-2026, illustrative):
  Input:  56,000 × $3/1M = ~$0.17
  Output: ~1,500 × $15/1M = ~$0.02
  Total per run: ~$0.19 uncached
  With prompt caching (0.1× reads on the shared prefix): ~$0.06–0.07

Multi-agent (4 subagents, orchestrator), ~15× a chat turn:
  ~$0.70–0.80 per task uncached
  At 100,000 tasks/day: $70,000–80,000/day uncached
  Caching the shared, growing prefix cuts the input bill ~3×
  (~$0.12 saved per single-agent run = ~$12,000/day at 100k tasks)

3. No-progress detector. Compare the last N tool calls. If they are identical (same tool name, same inputs), the agent is oscillating. Stop it. If the last N tool results all contain error strings, the agent is stuck on a broken tool. Stop it and escalate.

def check_no_progress(messages: list, window: int = 3) -> bool:
    """Returns True if the last `window` tool calls are identical."""
    tool_calls = [
        (block.name, str(block.input))
        for msg in messages
        if msg["role"] == "assistant"
        for block in (msg["content"] if isinstance(msg["content"], list) else [])
        if hasattr(block, "type") and block.type == "tool_use"
    ]
    if len(tool_calls) < window:
        return False
    return len(set(tool_calls[-window:])) == 1

Why error rates compound so badly

Here is the number that makes every agent demo look better than it is.

{ "type": "error-compound", "accuracy": 0.85, "steps": 10, "title": "Per-step accuracy compounds fast" }

At 85% per-step accuracy — which is good, better than many production systems — a 10-step workflow succeeds end-to-end only about 20% of the time. The math is 0.85^10 ≈ 0.197. At 95% accuracy per step, a 10-step workflow succeeds about 60% of the time. To hit 90% end-to-end success on a 10-step task, you need 99% per-step accuracy.

This is why the single most important investment in agent reliability is not picking a better model for the orchestrator — it's shortening the chains. Three 5-step pipelines with checkpoints between them will reliably outperform one 15-step autonomous chain, because you get to validate the intermediate results and abort early if something is wrong. The agent evaluation article covers the measurement methodology; the loop design principle is: keep chains short and verify at boundaries.

What actually goes in the context

The context window is the agent's only working memory within a loop. On each iteration, the model can see exactly what is in its messages array and nothing else. This sounds obvious but has non-obvious consequences.

{ "type": "context-window", "window": 128000, "title": "Agent context budget across iterations" }

At the start of a 10-iteration run, your context looks roughly like this:

Segment	Tokens (approx)	Notes
System prompt	500–2,000	Static; prime candidate for caching
Tool definitions	290–800 per tool set	290-800 tokens just for Anthropic's tool-use system prompt injection
User task	50–500
Per-iteration history	~800 per iteration	Tool call + result, growing linearly
Output reserve	1,000–4,096	Must keep headroom for the next response

After 10 iterations at 800 tokens each, you have consumed roughly 2,000 (static) + 8,000 (history) + 1,000 (output reserve) = ~11,000 tokens minimum. On a 128k context model that's fine. On a 32k model with verbose tool results, you'll hit the window in fewer than 20 iterations. The context window management article covers compression strategies; the budget math above tells you when to start caring.

The static segments — system prompt, tool definitions — are the prime candidates for prompt caching. Tool definitions especially: they don't change between iterations, but they get re-sent on every call. On Anthropic, a cached prefix costs 0.1× the normal input price after the first call. On a 100k-task/day deployment, caching the 2,000-token static prefix saves real money (see the cost block above).

The tool call round-trip in practice

A few mechanics that bite people in production:

Parallel tool calls. Both Anthropic and OpenAI allow the model to emit multiple tool calls in a single response. Execute them concurrently and send all results back in one turn — do not send each result as a separate turn. The model expects all the results it requested before it reasons again.

# Parallel execution of multiple tool calls
import asyncio

async def execute_tools_parallel(tool_calls, executor):
    tasks = [executor(tc.name, tc.input) for tc in tool_calls]
    results = await asyncio.gather(*tasks, return_exceptions=True)
    return [
        {
            "type": "tool_result",
            "tool_use_id": tc.id,
            "content": str(r) if not isinstance(r, Exception) else f"Error: {r}",
            "is_error": isinstance(r, Exception),
        }
        for tc, r in zip(tool_calls, results)
    ]

Tool errors should go back to the model, not crash your application. When a tool fails, set is_error: true in the result and include a structured error message. The model will often adapt — try a different parameter, use a fallback tool, or ask for clarification. Hard-crashing on first tool failure wastes partial progress and produces worse user experience than letting the model recover.

But "let the model retry" only applies to read-only tools. get_order_status can run five times with no harm; issue_refund cannot. Classify every tool as read-only or side-effecting before wiring it into a loop. For side-effecting tools, three rules: give each call an idempotency key so a retried request is deduplicated server-side; never auto-retry on a timeout, because the first call may have succeeded — check the resulting state instead; and gate irreversible actions (refunds, outbound emails, deletes) behind a human or policy approval step. Host-side execution exists precisely so you can hold a tool_use block, validate or approve it, and only then run it — use that hook.

Refusals are structured objects, not empty strings. On OpenAI's API, when the model declines to produce a structured output, it returns a refusal field in the message. Code that blindly parses message.content will crash. Always check message.refusal (OpenAI) or inspect stop_reason (Anthropic) before extracting content.

What breaks

Agent loops fail in a small number of ways, most of them predictable:

Oscillation. The agent calls the same tool with the same arguments repeatedly. Usually caused by: the tool returns an error the model doesn't understand, the model's belief about the task state is wrong, or the tool result contradicts what the system prompt implies is possible. Fix: no-progress detector (above) + structured error messages from tools.

Context rot. After many iterations, the prompt contains so much overlapping and conflicting information that the model's responses degrade. Symptoms: the model starts ignoring earlier instructions, "forgets" constraints stated in the system prompt, or produces responses inconsistent with the conversation history. Fix: compress or summarize earlier turns when the context exceeds ~60% of the window.

Silent wrong answers. The model emits end_turn with a confident-sounding answer that is factually wrong or incomplete. No exception is raised. No error is logged. The user gets garbage. This is the hardest failure mode to catch because it is invisible to your application. Fix: output validation on the final response — either a Pydantic/Zod schema check, a secondary LLM evaluator, or explicit post-task assertions specific to your task.

Runaway cost. No iteration cap, verbose tool results, a task that keeps spawning subtasks. Fix: iteration cap + token budget enforced before each LLM call, not as a soft preference but as a hard gate.

Tool hallucination. The model calls a tool that does not exist, or passes parameters that are not in the schema. This is rarer with good schema design but does happen. Fix: validate tool call names and parameters against your registered tool list before execution; return a structured error ("unknown tool") if the model invents one.

For a comprehensive taxonomy of failure modes and the evaluation methodology to catch them systematically, see Agent Evaluation and Why Agents Fail in Production.

How the agent loop connects to everything else

The loop described here is the foundation. Every other concept in this section is a layer on top of it:

Tool and function calling covers how tool schemas are designed and how the model uses them to decide what to call — the mechanism that turns "Act" into a structured API call.

Agent Memory covers what to do when the context window isn't enough — episodic stores, semantic vector databases, and procedural caches that persist knowledge across loop restarts.

ReAct, Reflexion, and Planning Patterns covers the prompting strategies — ReAct's interleaved thought-action traces, Reflexion's self-critique loop, and Plan-and-Execute's upfront decomposition — that change what the model does in the Reason and Plan stages.

Multi-Agent Orchestration covers what happens when you run multiple agent loops in coordination — the orchestrator-worker pattern, dynamic handoffs, and the state management that keeps context from getting lost between agents.

Model Context Protocol covers the standardized protocol (adopted across OpenAI, Anthropic, Google, and Microsoft by mid-2025) that separates tool servers from agent logic, so the same tool integrations work across different agent frameworks.

flowchart TD
    LOOP["Agent Loop\n(this article)"]
    TOOLS["Tool Calling\n& Schemas"]
    MEMORY["Agent Memory\n(4 tiers)"]
    PLANNING["ReAct / Reflexion\n/ Planning Patterns"]
    MULTI["Multi-Agent\nOrchestration"]
    MCP["Model Context\nProtocol"]
    EVAL["Agent Evaluation\n& Failure Modes"]

    LOOP --> TOOLS
    LOOP --> MEMORY
    LOOP --> PLANNING
    LOOP --> MULTI
    TOOLS --> MCP
    MULTI --> EVAL
    LOOP --> EVAL

    style LOOP fill:#a855f7,color:#fff
    style TOOLS fill:#0e7490,color:#fff
    style MEMORY fill:#0e7490,color:#fff
    style PLANNING fill:#0e7490,color:#fff
    style MULTI fill:#0e7490,color:#fff
    style MCP fill:#0e7490,color:#fff
    style EVAL fill:#ff2e88,color:#fff

The decision in practice

Should you build an agent loop, or just chain a few LLM calls?

The honest answer: chain LLM calls first. A fixed three-call pipeline — classify the request, retrieve relevant data, generate the response — is predictable, cheap, and easy to test. Add a loop when the task requires a number of steps that you cannot enumerate upfront, or when the sequence depends on the output of prior steps in a way you can't hardcode.

When you do build a loop, start with a single agent before coordinating multiple. Anthropic's explicit guidance is to add agent complexity only when simpler solutions fall short. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, and in practice most multi-agent failures trace back to the same root cause: single-step reliability was never measured before multi-agent complexity was added.

The agent loop is not magic. It's a while loop with an LLM inside it. The model is good at reasoning about what to do next; it is not good at knowing when to stop, managing its own costs, or recovering from tool failures without explicit structure. Your job as the engineer is to provide that structure. Set the guards. Measure the per-step accuracy before you measure anything else. And get the termination logic right before you worry about which planning strategy the model uses.

The loop is simple. The guards are what make it work.