The tool-use round trip: agentic loops, stop reasons, and parallel calls

The on-call alert came at 2 a.m. An agentic customer-service workflow had been running for four hours, consumed $340 in API tokens, and produced no output. The model kept calling search_order_history, getting results, calling search_order_history again with marginally different parameters, and looping. No iteration cap. No token budget. No check to see if the conversation had grown past 60k tokens, burying hours of tool results in the middle of the context — the region models attend to least reliably. The fix took 20 minutes. The post-mortem took two days.

That incident is not unusual. The tool-use loop is one of the simplest protocols in AI engineering and one of the most reliably misimplemented. This article covers the round trip in enough detail that you can write a production-safe loop on the first try.

The protocol, precisely

A tool-calling exchange is a structured conversation. The model is not calling your functions — it is telling you what to call. You call them. You tell the model the results. The model decides what to do next.

Here is what each turn looks like at the API level on Anthropic:

import anthropic

client = anthropic.Anthropic()

tools = [
    {
        "name": "get_weather",
        "description": "Get current weather for a city. Returns temperature in Celsius and conditions.",
        "input_schema": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name, e.g. 'Tokyo'"}
            },
            "required": ["city"]
        }
    }
]

messages = [{"role": "user", "content": "What's the weather in Tokyo and Berlin?"}]

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    tools=tools,
    messages=messages
)

print(response.stop_reason)   # "tool_use"
print(response.content)
# [TextBlock(text="I'll check the weather..."), ToolUseBlock(id="toolu_01", name="get_weather", input={"city": "Tokyo"}), ToolUseBlock(id="toolu_02", name="get_weather", input={"city": "Berlin"})]

The model returned stop_reason: "tool_use" and two ToolUseBlock objects in response.content. It has not yet generated a final answer. It is waiting for results.

The equivalent on OpenAI uses finish_reason: "tool_calls" and the tool calls live in response.choices[0].message.tool_calls:

from openai import OpenAI

client = OpenAI()

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"}
                },
                "required": ["city"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="gpt-4o",
    tools=tools,
    messages=[{"role": "user", "content": "Weather in Tokyo and Berlin?"}]
)

choice = response.choices[0]
print(choice.finish_reason)          # "tool_calls"
print(choice.message.tool_calls)     # [ChatCompletionMessageToolCall(...), ...]

Different field names, same idea. The thing that trips engineers up is what to do next.

The full loop

The complete loop is short enough to fit on a screen:

import anthropic
import concurrent.futures
import json

client = anthropic.Anthropic()

def run_tool(tool_name: str, tool_input: dict) -> str:
    """Dispatch to the actual tool implementation."""
    if tool_name == "get_weather":
        # Real implementation would call a weather API
        return json.dumps({"temperature": 22, "conditions": "Partly cloudy"})
    raise ValueError(f"Unknown tool: {tool_name}")

def agentic_loop(messages: list, tools: list, max_iterations: int = 15) -> str:
    iteration = 0

    while iteration < max_iterations:
        response = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=4096,
            tools=tools,
            messages=messages,
        )

        # Always append the assistant response before doing anything else
        messages.append({"role": "assistant", "content": response.content})

        if response.stop_reason == "end_turn":
            # Extract final text and return
            for block in response.content:
                if hasattr(block, "text"):
                    return block.text
            return ""  # model finished with no text (unusual)

        if response.stop_reason != "tool_use":
            raise RuntimeError(f"Unexpected stop_reason: {response.stop_reason}")

        # Collect all tool_use blocks
        tool_use_blocks = [b for b in response.content if b.type == "tool_use"]

        # Execute them in parallel
        tool_results = []
        with concurrent.futures.ThreadPoolExecutor() as executor:
            futures = {
                executor.submit(run_tool, b.name, b.input): b
                for b in tool_use_blocks
            }
            for future, block in futures.items():
                try:
                    result = future.result(timeout=30)  # per-tool timeout
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": result,
                    })
                except concurrent.futures.TimeoutError:
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": "Error: tool timed out after 30 seconds",
                        "is_error": True,
                    })
                except Exception as exc:
                    # Return the error as a string — the model can reason about it
                    tool_results.append({
                        "type": "tool_result",
                        "tool_use_id": block.id,
                        "content": f"Error: {exc}",
                        "is_error": True,
                    })

        # All results go in a single user turn
        messages.append({"role": "user", "content": tool_results})
        iteration += 1

    raise RuntimeError(f"Agentic loop hit iteration limit ({max_iterations})")

A few things this code gets right that naive implementations miss:

Append before branching. The assistant message — including the tool_use blocks — must be appended to messages before you execute any tools and certainly before you send the next API call. If you skip this, the model receives a user message containing tool results with no corresponding assistant message requesting them, which violates the conversation structure and causes API errors or confusing model behavior.

Execute all tool blocks before re-calling. If the response contains two tool_use blocks, both results must be present in the next user turn. Sending one result and calling the model is a protocol error on Anthropic; on OpenAI it similarly breaks the expected message structure. Run them concurrently and collect all results before assembling the user turn.

Return structured errors. When a tool fails, do not raise an exception in the loop — return the error as a string in the tool_result content with is_error: true. The model can read that error and decide whether to retry with different parameters, apologize to the user, or escalate. An exception that crashes the loop throws away all intermediate progress.

Time out individual tools. Iteration and token budgets are blind to a tool that simply hangs — a wedged HTTP call or a slow database query stalls the loop forever while both counters sit still. The future.result(timeout=30) call converts a hung tool into a structured timeout error the model can reason about, exactly like any other tool failure. The four-hour incident that opened this article burned most of its wall-clock time this way.

The iteration cap is not optional. Without it, a stuck model will loop until it either hits the API's max context length (which raises an error with no useful output) or until you get the 2 a.m. alert.

Parallel calls: the performance multiplier

When a user asks for weather in Tokyo and Berlin simultaneously, a well-instructed model should return both get_weather requests in a single response rather than calling Tokyo, waiting for results, then calling Berlin. That is parallel tool calling, and it makes a real latency difference.

Sequential: latency = T(Tokyo) + T(Berlin) = 200ms + 180ms = 380ms
Parallel:   latency = max(T(Tokyo), T(Berlin)) = max(200ms, 180ms) = 200ms

At three or more concurrent tools the savings compound. The parallel pattern also reduces token cost: you pay for one round trip through the model instead of N.

{ "type": "agent-loop", "scenario": "weather-trip", "title": "One agent turn, step by step: two weather calls in one response" }

Models do not always use parallel calls even when they could. You can encourage parallelism by telling the model explicitly in the system prompt or tool descriptions: "When multiple tool calls are independent, request them all in the same response." Some teams add this as a standing system-prompt instruction:

When you need to call multiple tools and the calls are independent (one result
does not affect another's parameters), request all of them in the same response
turn rather than sequentially. This reduces latency and total token cost.

The OpenAI strict-mode incompatibility

OpenAI's Structured Outputs (strict: true) uses constrained decoding at the server to guarantee schema conformance. That constraint mechanism does not extend to multiple parallel tool call blocks in a single response — and here is the trap: the API does not reject the combination. The request succeeds, and parallel tool calls may simply not match your supplied schemas. The misconfiguration is silent; nothing catches it for you.

The fix is a single flag you must set yourself:

response = client.chat.completions.create(
    model="gpt-4o",
    tools=tools,
    tool_choice="auto",
    parallel_tool_calls=False,   # required when strict: true
    messages=messages,
)

The tradeoff: you lose the latency benefit of parallelism in exchange for schema guarantees on each individual tool call. For most extraction workflows where correctness matters more than speed, that is the right call. For agentic flows where the model is orchestrating many fast tools, disabling strict mode and validating results yourself is often the better path. See Building Reliable Tool-Using Systems for validation patterns.

Token cost of tool definitions

Every API call with tool definitions attached carries token overhead that most engineers underestimate. Anthropic injects a system prompt containing the tool definitions and instructions for how to use them. That overhead is:

Anthropic tool use overhead (illustrative, as of mid-2026):
  No tools:               0 extra tokens
  tools only (auto):    ~290 tokens
  tools + tool_choice:  ~804 tokens depending on model and setting

At $3/MTok for claude-sonnet-4-5 input tokens:
  10,000 API calls/day × 500 token overhead avg = 5M tokens/day
  5M × $3/MTok = $15/day just in tool-definition overhead
  $450/month for tokens the user never sees

This is why Anthropic's Programmatic Tool Calling (beta header advanced-tool-use-2025-11-20) matters for tool-heavy workflows. Instead of emitting tool_use blocks one at a time, Claude writes Python that orchestrates the calls inside a code-execution environment — and the intermediate tool results are consumed there rather than re-entering the model's context, with fewer inference passes along the way. Anthropic reports roughly 37% lower token consumption on its internal evals.

For tool-heavy applications, the math on tool count versus call frequency is worth running before you commit to an architecture. The definition-overhead problem specifically is what Anthropic's separate Tool Search Tool (2025) addresses: Claude discovers tools from a catalog on demand rather than receiving all definitions upfront, which collapses per-request overhead to near zero for large catalogs.

What breaks: failure modes

{ "type": "error-compound", "accuracy": 0.95, "steps": 10, "title": "Compounding failure in a 10-step tool chain" }

Context rot

The most insidious failure mode in production agents. As the conversation grows — system prompt, user query, tool definitions, assistant messages, tool results, more assistant messages, more tool results — earlier tool results and instructions accumulate in the middle of a long context, exactly where models attend least reliably. This is the lost-in-the-middle effect: attention is strongest at the beginning and end of the window and sags in between, and a growing agent conversation keeps shoving its history into that sag. After roughly 10-20 tool call turns in a naive implementation, you will see:

• The model re-calling tools it already called with the same parameters
• Instructions from the system prompt being ignored
• Lower-quality reasoning because the model is "reading" a very long document and missing context from earlier sections

Active context management is the fix. Between turns, or after every N tool calls, summarize the accumulated tool results into a compact form and replace the raw results with the summary. Libraries like LangGraph have this built in as a checkpointing and pruning operation; rolling your own requires explicit handling.

For the mechanics of what to prune versus compress and when, see the context window management article.

Hallucinated tool calls

The model invents a tool name that does not exist in your tool list, or calls a real tool with a parameter value it invented rather than extracted from context. Both are silent failures if you are not validating tool names and argument schemas before execution.

Guard against hallucinated tool names by explicitly comparing the tool name in each tool_use block against your registered tool list. If the model calls search_orders_v2 and you only have search_orders, return a structured error explaining that the tool does not exist rather than raising an exception.

Hallucinated parameter values — a fabricated order ID, a made-up date — require semantic validation beyond schema checking. The schema confirms the shape; your code confirms the business logic. A well-formed tool call with an ID that does not exist in your database is still wrong.

Refusals

OpenAI Structured Outputs can return a refusal object instead of the expected JSON:

choice = response.choices[0]
if choice.message.refusal:
    # The model declined to produce tool call arguments
    raise ValueError(f"Model refused: {choice.message.refusal}")
# Only parse tool_calls after this check
tool_calls = choice.message.tool_calls

Code that blindly accesses message.tool_calls without checking message.refusal first will raise a NullPointerException-class error when a refusal occurs. Refusals happen when the model determines that executing the tool call would violate its guidelines given the context — for example, a tool that sends emails called in a context that looks like a spam campaign.

The runaway loop

Without an iteration cap, a model that cannot make progress will loop indefinitely. This can happen when:

• A required external service is returning errors the model cannot interpret
• The task is genuinely impossible given the available tools
• The model is in a reasoning rut, trying slight variations of the same tool call

Add a token budget alongside the iteration cap:

def agentic_loop_with_budget(
    messages: list,
    tools: list,
    max_iterations: int = 15,
    max_total_tokens: int = 50_000,
) -> str:
    iteration = 0
    total_tokens = 0

    while iteration < max_iterations and total_tokens < max_total_tokens:
        response = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=4096,
            tools=tools,
            messages=messages,
            system=(
                f"You have {max_iterations - iteration} turns remaining and "
                f"approximately {max_total_tokens - total_tokens} tokens remaining. "
                "If you cannot complete the task within budget, summarize what you found."
            ),
        )

        total_tokens += response.usage.input_tokens + response.usage.output_tokens
        # ... rest of loop

Telling the model its remaining budget is more effective than silently cutting it off. The model can then produce a useful partial answer or a "I could not complete X because Y" response rather than an abrupt termination.

One caveat on placement: interpolating a per-turn counter into the system prompt busts prompt caching from token zero on every call — the system prompt is the cache-stable prefix, and changing it invalidates everything after it. If you use prompt caching (and for agent loops you should), put the budget notice at the end of the conversation instead: append it as a short text block alongside the tool results in the trailing user turn. Same signal to the model, and the cached prefix survives. Only bake it into the system prompt if you have consciously decided the cache loss is worth it.

Worked latency and cost estimate

Suppose you are building an agentic order-lookup workflow with an average of 4 tool calls per user request:

Assumptions (illustrative):
  Model: claude-sonnet-4-5 (input $3/MTok, output $15/MTok, mid-2026 pricing)
  Avg input tokens per turn: 2,000 (system + history + tool defs + results)
  Avg output tokens per turn: 200 (tool calls + brief reasoning)
  4 tool turns + 1 final turn = 5 model calls per request
  Tool execution latency: 50ms avg (DB lookups, internal API)
  Model API latency: 800ms avg per call

Per-request cost:
  Input:  5 × 2,000 = 10,000 tokens × $3/MTok = $0.030
  Output: 5 × 200   = 1,000 tokens  × $15/MTok = $0.015
  Total per request: ~$0.045

Per-request latency (parallel where possible):
  5 model calls serial: 5 × 800ms = 4,000ms
  Tool calls parallel within each turn: adds 50ms per turn
  Total: ~4,250ms end-to-end

At 1,000 requests/day:
  Daily cost: 1,000 × $0.045 = $45/day = ~$1,350/month
  
If tool defs add 500-token overhead per call (tool_choice set):
  Extra: 5 turns × 500 tokens × $3/MTok × 1,000 req = $7.50/day
  → Worth investigating Programmatic Tool Calling if tool set is large

The model API latency dominates. At 5 sequential model calls, you are at 4+ seconds even before tool execution. If that is too slow for your use case, the options are: reduce the number of required tool turns by giving the model more information upfront, use a faster model tier for intermediate turns and reserve the full-capability model for the final synthesis, or cache tool results across requests when the same query repeats.

The stop-reason decision tree

flowchart TD
    A["Call model API"] --> B{"stop_reason?"}
    B -->|"end_turn / stop"| C["Extract final text, return"]
    B -->|"tool_use / tool_calls"| D["Collect all tool_use blocks"]
    B -->|"max_tokens"| E["Response truncated — either increase max_tokens or surface partial result"]
    B -->|"refusal"| F["Handle refusal — log it, return error to user"]
    D --> G["Execute all tools concurrently"]
    G --> H["Append assistant message + all tool_results to messages"]
    H --> I{"Budget check:\niterations < max AND\ntokens < max?"}
    I -->|Yes| A
    I -->|No| J["Surface partial result + reason for stopping"]
    style C fill:#15803d,color:#fff
    style E fill:#00e5ff,color:#0a0a0f
    style F fill:#ff2e88,color:#111
    style J fill:#ffaa00,color:#0a0a0f

max_tokens truncation is the most commonly forgotten case. If the model's output is cut off mid-tool-call, you get a malformed tool_use block that will fail JSON parsing. The right response is not to retry blindly — increase max_tokens for your tool-heavy requests or inspect whether the model is over-thinking intermediate steps.

Connecting tool calls to the broader agent architecture

The round trip described here is the primitive. The Agent Loop article covers the perceive-reason-act structure built on top of it — how memory, planning, and environment observation compose into something that resembles autonomous behavior. Designing Tool Schemas That Models Actually Understand covers the upstream decision: how your tool descriptions and parameter types shape whether the model calls the right tool in the first place.

For the downstream layer — what to do when the tool call returns a bad result, when to retry versus escalate, and how to build observability into each turn — Building Reliable Tool-Using Systems goes deeper on production patterns.

Multi-agent systems add another layer: tool calls that invoke other agents, which run their own inner loops. The same protocol applies recursively, but budget management becomes critical — a parent agent with a 15-iteration cap that spawns a child agent with its own 15-iteration cap can produce 225 model calls before either limit fires. See Multi-Agent Orchestration Patterns for how to handle nested budget accounting.

Anthropic vs OpenAI: the differences that matter

flowchart LR
    subgraph OpenAI["OpenAI"]
        OFC["finish_reason:\n'tool_calls'"]
        OTC["message.tool_calls\n[{id, function.name,\nfunction.arguments}]"]
        ORES["role: 'tool'\ntool_call_id: id\ncontent: result_string"]
        OSTM["strict: true\nrequires\nparallel_tool_calls: false"]
    end
    subgraph Anthropic["Anthropic"]
        ASR["stop_reason:\n'tool_use'"]
        ATU["content blocks:\nToolUseBlock(id, name, input)"]
        ARES["role: 'user'\ncontent: [{type: tool_result,\ntool_use_id: id}]"]
        APTR["Programmatic Tool Calling\nbeta header available"]
    end
    style OFC fill:#0e7490,color:#fff
    style ASR fill:#a855f7,color:#fff
    style OSTM fill:#00e5ff,color:#0a0a0f
    style APTR fill:#15803d,color:#fff

The structural difference that catches most people: on OpenAI, tool results go back as role: "tool" messages — separate messages, one per result. On Anthropic, tool results go in a role: "user" message as a list of tool_result content blocks. Building an abstraction layer that handles both shapes is straightforward but must be done explicitly; the naive direct mapping of one API's shape to the other silently produces malformed conversations.

The input field on Anthropic's ToolUseBlock is already a parsed Python dict. On OpenAI, function.arguments is a JSON string — you must call json.loads() before using the arguments. Another silent failure mode when porting code between providers.

The decision in practice

If you are building the loop for the first time, start with these defaults:

• max_iterations = 10 for extraction or lookup tasks; 15-20 for research or planning tasks
• Token budget: 50,000-100,000 total tokens across all turns for a single request
• Wall-clock budget: a per-tool timeout (future.result(timeout=30)) plus a total deadline of 2-5 minutes per request — a hung tool is invisible to both the iteration and token budgets
• Always check stop reason before touching any other field in the response
• Run parallel tool blocks with a thread pool; do not serialize what the model has already parallelized for you
• Return structured errors from failed tools, not Python exceptions
• On OpenAI: decide between strict mode and parallel calls before writing any other code; the API will not error if you enable both, it just stops guaranteeing your schemas

The loop itself is perhaps 40 lines of code. The discipline around that loop — the budget tracking, the error handling, the context pruning — is where production-ready agents actually differ from demos. The 2 a.m. incident that opened this article could have been prevented by a three-line iteration cap. Start there.