# Building Reliable Tool-Using Systems: Patterns From Production > Production patterns for reliable LLM tool use: loop budgeting, validation layers, idempotency, fallbacks, and observability that prevent runaway agents. Canonical: https://ironclad.academy/ai-engineering/articles/reliable-tool-using-systems-production-patterns | Difficulty: advanced | Topics: Tool Use, Agents, Reliability, Production Covers: OpenAI, Anthropic ## Key takeaways - Mostly deterministic workflows with small, bounded agent micro-loops beat fully autonomous agents on reliability — a pattern confirmed repeatedly in production deployment analyses. - Schema conformance from constrained decoding is not semantic correctness — you still need a Pydantic/Zod validation layer that enforces business rules after parsing. - Return structured, typed errors from tool execution so the model can reason about whether to retry, escalate, or abort — empty error strings cause hallucinated retries. - Set explicit iteration caps and cumulative token budgets before the loop starts; hitting either limit should return a structured failure, not raise an exception. - Make every tool call idempotent with caller-generated idempotency keys — the model will retry, sometimes identically, and your state layer must survive it. - Trace each tool_use block with its input, output, latency, and token cost separately; without per-tool observability you cannot localize failures in a multi-step chain. > **In one line:** Production tool-using systems need loop budgets, semantic validation layers, structured error returns, idempotent tool implementations, and per-tool observability — the model doing the right thing is necessary but not sufficient. **The problem.** An agentic loop that works in a demo has three hidden properties: the happy path never throws exceptions, the context fits comfortably in the window, and the test run terminates in under five steps. Production has none of those properties. Tool calls fail with confusing errors, conversations grow until earlier results drop off the window, and a model that doesn't know how to stop will spend $40 in tokens trying. Schema-valid JSON that contains a hallucinated order ID will pass every structural check and blow up at the database. **What this article covers.** The production patterns that close the gap — loop budget design, structured error contracts between tools and model, validation beyond schema conformance, idempotency, fallback strategies, and observability. The [tool-use round trip mechanics](/ai-engineering/articles/tool-use-round-trip-agentic-loop), [schema design](/ai-engineering/articles/designing-tool-schemas-llm-agents), and [constrained decoding internals](/ai-engineering/articles/structured-outputs-constrained-decoding) are covered in their own articles; this one is about what happens after all that and before the system works reliably. **Key points.** - Bounded, deterministic-first designs beat open-ended agentic autonomy on reliability at production scale. - Schema validation and semantic validation are separate layers; constrained decoding handles the first, you must implement the second. - Structured error returns from tool execution are the difference between a model that recovers gracefully and one that hallucinates a successful result. - Idempotency keys protect against model-generated duplicate calls and network-failure retries. - Per-tool tracing (input, output, latency, tokens) is the only way to localize failures in multi-step chains. ``` Back-of-the-envelope: cost of an unbudgeted loop Suppose an agent loop uses a mid-tier model at $3/M input + $12/M output tokens. Each iteration costs ~2,000 input tokens (context grows) + ~500 output tokens. → per-iteration cost ≈ (2000 × $3 + 500 × $12) / 1,000,000 = ~$12/1000 = $0.012 At 10 iterations: $0.12 — reasonable. At 50 iterations (stuck loop): $0.60 — alarming at scale. At 200 iterations (runaway): $2.40 per single request. With 1,000 daily active sessions each triggering one runaway: 2,400 USD/day in runaway loops alone. A 10-iteration cap costs you $0 in normal cases and caps the worst case at $0.12/session. ``` ```mermaid flowchart TD A[User Request] --> B[Agent Orchestrator] B --> C{"Budget check\niter < max_iter\ntokens < max_tokens"} C -- over budget --> FAIL["Return structured\nfailure + partial result"] C -- within budget --> D[Call LLM] D --> E{stop_reason} E -- end_turn --> DONE[Return final answer] E -- tool_use --> F["Validate tool args\nPydantic / Zod"] F -- invalid --> RETRY["Return schema error\nto model"] F -- valid --> G[Execute tool] G --> H{Tool result} H -- success --> I["Append tool_result\nback to context"] H -- error --> J["Return structured\nerror to model"] I --> B J --> B RETRY --> B style FAIL fill:#ff2e88,color:#111 style DONE fill:#15803d,color:#fff style RETRY fill:#ffaa00,color:#111 style J fill:#ffaa00,color:#111 ``` **Key design decisions.** | Decision | Recommended default | When to deviate | | --- | --- | --- | | Max iterations per loop | 10 for focused tasks, 30 for research workflows | Increase only with per-step token accounting | | Error return format | Structured JSON with `error_type`, `message`, `retryable` | Simple tasks can use plain-text errors | | Idempotency | Caller-generated key on every write tool | Skip for pure read tools | | Validation order | Schema first (Pydantic), then semantic | Skip semantic only for read-only tools with no side effects | | Fallback model | Same-family smaller model on 503/429 | Cross-family fallback needs extra schema compatibility testing | | Tracing granularity | One span per tool_use block | Aggregate spans only after you have working per-tool traces | **If you have 60 seconds, say this.** "Tool-using systems fail in production in four predictable ways: runaway loops burn unbounded tokens, schema-valid outputs contain semantically wrong values, tool errors come back as empty strings so the model retries blindly, and duplicate calls create conflicting state. I fix the first with explicit iteration and token budgets set before the loop starts. The second needs a Pydantic or Zod layer after schema parsing that enforces business rules — not just types. The third needs a structured error contract: every tool returns a typed error object with `error_type` and `retryable`. The fourth needs idempotency keys on all write tools. On top of that, I trace every tool call with input, output, latency, and tokens so I can actually localize a failure in a five-step chain." A team at a fintech company shipped a credit-decision agent in September 2025. In the demo, the agent called three tools — fetch customer profile, pull credit bureau data, run eligibility rules — and returned a clean decision in four seconds. In production, a non-existent customer ID (a typo in the intake form) caused the credit bureau tool to return a `null` body. The agent didn't get a structured error; it got an empty string. So it called the tool again. And again. Fourteen times over 22 seconds and $1.80 in tokens before a hard HTTP timeout cut it off. The decision never came back; the customer got a 504. Their incident postmortem had two action items: structured error contracts and an iteration cap. Those two things cover roughly 60% of the production failure modes this article is about. ## Why demos lie A working demo has a specific failure mode: it selects for the happy path. The model picks the right tool, fills in correct arguments, the tool returns a clean response, the model says something sensible. You run it five times, it works five times, you ship. Production traffic is not the happy path. Users submit malformed input. Backend services return 429s and 503s during peak load. Network timeouts produce no response at all. A tool returns a valid object but with a foreign key that doesn't exist in your database. The model, faced with an ambiguous situation and no hard stopping condition, does the thing it was trained to do: it keeps trying. An [industry analysis of ~1,200 production LLM deployments](https://www.zenml.io/blog/what-1200-production-deployments-reveal-about-llmops-in-2025) (ZenML, mid-2025) found a consistent pattern across successful systems: **mostly deterministic workflows with small, bounded agent micro-loops** outperformed fully autonomous agents on reliability at any scale. The winning architecture looks like an orchestrated pipeline where each step is deterministic Python that might invoke a small 3–5 step agent sub-loop with tight guardrails — not a single agent given a task and left to run until it decides it's done. The [12-Factor Agents manifesto](https://github.com/humanlayer/12-factor-agents) (2025), which has become something of a community consensus document for production agentic systems, states the same principle differently: treat each agent turn as stateless, externalize all state to durable storage, and make tool calls idempotent. These aren't aspirational — they're requirements if you want the system to recover from the failures that will definitely happen. ## Budgeting the loop The single most important reliability primitive for an agentic loop is also the simplest: a hard cap on iterations and a cumulative token budget, checked at the top of each iteration before calling the model. DoorDash published their [agentic platform design](https://careersatdoordash.com/blog/beyond-single-agents-doordash-building-collaborative-ai-ecosystem/) in 2025 and called this "budgeting the loop." The implementation is straightforward: ```python from anthropic import Anthropic client = Anthropic() def run_agent( messages: list[dict], tools: list[dict], max_iterations: int = 10, max_tokens: int = 50_000, ) -> dict: iteration = 0 total_tokens_used = 0 while True: if iteration >= max_iterations: return { "status": "budget_exceeded", "reason": "max_iterations", "iterations_used": iteration, "tokens_used": total_tokens_used, "partial_messages": messages, } if total_tokens_used >= max_tokens: return { "status": "budget_exceeded", "reason": "max_tokens", "iterations_used": iteration, "tokens_used": total_tokens_used, "partial_messages": messages, } response = client.messages.create( model="claude-sonnet-4-5", max_tokens=4096, tools=tools, messages=messages, ) total_tokens_used += response.usage.input_tokens + response.usage.output_tokens iteration += 1 if response.stop_reason == "end_turn": return { "status": "success", "result": response.content, "iterations_used": iteration, "tokens_used": total_tokens_used, } if response.stop_reason == "tool_use": tool_results = execute_tools(response.content) messages = messages + [ {"role": "assistant", "content": response.content}, {"role": "user", "content": tool_results}, ] else: # stop_reason == "max_tokens" or unexpected return { "status": "failed", "reason": f"unexpected_stop_reason:{response.stop_reason}", "iterations_used": iteration, "tokens_used": total_tokens_used, } ``` The return type matters as much as the logic. A budget-exceeded result is not an exception — it's a structured response with a `status`, the reason, and the partial message history. The caller can decide to log it, return a partial result to the user, or escalate to a human. Raising a `RuntimeError("loop budget exceeded")` gives the caller nothing useful. What values to set? For a focused task agent (answer one specific question, fill one form), `max_iterations=10` and `max_tokens=50_000` are a reasonable starting point. For a research-style agent that may genuinely need to call many tools, `max_iterations=30` and `max_tokens=150_000` are more appropriate — but pair it with per-step cost accounting so you can see where the tokens are actually going. Start tight and loosen based on observed production data, not intuition. There's a second, less obvious reason to keep loops short: per-step errors compound. If each step in the chain succeeds 92% of the time — a decent number for a tool call that includes model reasoning, argument generation, and execution — a 10-step chain completes cleanly only 0.92¹⁰ ≈ 43% of the time. That's why the iteration cap and the per-step validation layers below aren't independent fixes; the cap limits how many chances the chain has to fail, and validation raises the per-step number that gets exponentiated. [Interactive visualizer on the original page: error-compound — How per-step errors compound across an agentic loop] ## The two-layer validation problem Constrained decoding and strict-mode schemas — covered in depth in the [structured outputs article](/ai-engineering/articles/structured-outputs-constrained-decoding) — guarantee that the model's tool call arguments are syntactically valid and conform to the declared schema. That's layer one. Layer two is your problem. A tool call can be perfectly schema-valid and still wrong: - `order_id: "ORD-8472"` is a valid string but that order doesn't exist. - `quantity: -5` passes an `integer` type check but is nonsensical. - `delivery_date: "2024-01-15"` is a valid ISO 8601 date but it's in the past. - `user_id: "usr_999"` matches the pattern but belongs to a different tenant. None of these are caught by JSON Schema validation. They're all caught by a semantic validation layer — business-rule checks that run after schema parsing, before tool execution. In Python, Pydantic is the standard tool for combining both layers: ```python from pydantic import BaseModel, Field, field_validator, model_validator from datetime import date from typing import Annotated class CreateOrderArgs(BaseModel): customer_id: str = Field(pattern=r"^cust_[a-zA-Z0-9]+$") product_sku: str quantity: Annotated[int, Field(gt=0, le=1000)] delivery_date: date @field_validator("delivery_date") @classmethod def delivery_must_be_future(cls, v: date) -> date: if v <= date.today(): raise ValueError(f"delivery_date must be in the future, got {v}") return v @model_validator(mode="after") def validate_tenant_isolation(self) -> "CreateOrderArgs": # Check that customer_id belongs to the current tenant context # (tenant_id comes from the call context, not the model arguments) if not customer_exists_in_tenant(self.customer_id): raise ValueError( f"customer_id {self.customer_id!r} not found — " "check that the ID is correct and belongs to this account" ) return self ``` The error message in the `ValueError` matters. When validation fails, you return that message to the model as a structured error — and the model will use the text to decide what to try next. "customer_id not found" gives it something to work with. "Validation error" does not. The retry pattern for validation failures is to append the error as a user message and re-invoke: ```python def execute_tool_with_validation(tool_name: str, raw_args: dict) -> dict: validator = TOOL_VALIDATORS.get(tool_name) if validator is None: return {"error_type": "unknown_tool", "message": f"No tool named {tool_name!r}", "retryable": False} try: validated = validator(**raw_args) except ValidationError as e: return { "error_type": "validation_error", "message": str(e), "retryable": True, "hint": "Check the field values against the tool's parameter descriptions", } return run_tool_implementation(tool_name, validated) ``` A pattern worth internalizing: a smaller model with explicit schemas and a strong validation layer often ships more reliable function calling than a larger model trusted to get field values right on its own. The validation layer is doing real work — it's not just defensive programming. ## Structured error contracts The most common single fix that improves agentic reliability in production is changing tool execution errors from empty strings (or Python exceptions) to structured objects. When a tool returns `""` or `None` on failure, the model has no information. Its options are: assume success and continue, retry with identical arguments, or generate a plausible-sounding result. In practice it cycles through all three, none of which is correct. When a tool returns a structured error, the model can reason: this is a `not_found` error, it's `retryable: false`, I should tell the user I couldn't find that record. Every tool in a production system should return a result that fits one of two shapes: ```python # Success { "status": "success", "data": { ... } # tool-specific payload } # Error { "status": "error", "error_type": "not_found", # one of: not_found, permission_denied, # validation_error, rate_limited, timeout, # upstream_error, internal_error "message": "Order ORD-8472 not found in account acct_123", "retryable": False, "context": { # optional: machine-readable detail "searched_account": "acct_123", "order_id": "ORD-8472" } } ``` The `error_type` field is critical. A `rate_limited` error with `retryable: True` should prompt the model to wait and retry or tell the user to try again later. A `permission_denied` error with `retryable: False` should prompt it to escalate or report the issue. A `not_found` error might indicate a model hallucination that the model should acknowledge — "I apologize, I cannot locate that order ID" — rather than silently retry. ```mermaid sequenceDiagram participant O as Orchestrator participant M as Model participant T as Tool Layer participant V as Validator participant DB as Database O->>M: messages + tools M-->>O: tool_use: get_order(order_id="ORD-8472") O->>V: validate args V-->>O: valid O->>T: execute get_order("ORD-8472") T->>DB: SELECT WHERE id='ORD-8472' DB-->>T: 0 rows T-->>O: {status:"error", error_type:"not_found", retryable:false} O->>M: tool_result: {status:"error"...} M-->>O: "I was unable to locate order ORD-8472.\nCould you verify the order ID?" O-->>O: end_turn → return to user ``` ## Idempotency in tool execution The model will sometimes generate duplicate tool calls. It happens with parallel tool calls when the model is slightly uncertain, it happens on retry after an ambiguous response, and it happens when the calling code retries a failed API request and the tool gets executed twice. For read operations this is harmless. For writes — creating records, charging payment methods, sending notifications — it's a correctness problem. The standard fix is caller-generated idempotency keys. The orchestrator generates a stable key for each logical tool invocation and passes it with the tool call. The tool implementation deduplicates on that key: ```python import hashlib import json def make_idempotency_key(session_id: str, tool_name: str, args: dict) -> str: """Stable key for a logical tool invocation within a session.""" canonical = json.dumps({"session": session_id, "tool": tool_name, "args": args}, sort_keys=True) return hashlib.sha256(canonical.encode()).hexdigest()[:32] def execute_tool_idempotent( tool_name: str, args: dict, session_id: str, idempotency_store, # Redis or similar ) -> dict: key = make_idempotency_key(session_id, tool_name, args) cached = idempotency_store.get(key) if cached: # Redis stores bytes — serialize on write, deserialize on read return {"status": "success", "data": json.loads(cached), "idempotent_replay": True} result = run_tool_implementation(tool_name, args) if result["status"] == "success": idempotency_store.setex(key, 3600, json.dumps(result["data"])) # 1-hour TTL return result ``` One caveat on the key scheme: hashing `(session_id, tool_name, args)` means two *intentionally* identical writes in the same session — a user who legitimately places the same order twice — collapse into one execution, and the second caller silently gets a replay. If your domain allows distinct-but-identical operations, key on the model's `tool_use` block ID (unique per invocation) instead of the argument hash, and accept that you lose protection against the model regenerating the same call. The TTL matters. One hour is appropriate for most transactional operations. For long-running sessions or scheduled tasks, extend it to match the maximum session duration. For financial operations (charges, refunds), consider persisting the idempotency record in your database rather than Redis — you want durability, not just cache performance. One detail: only cache successful results. If a tool fails, don't cache the failure — the next call should try again. The exception is `retryable: False` errors with permanent semantics (a `not_found` for a resource that genuinely doesn't exist) — caching those prevents redundant database lookups for resources that won't suddenly appear. ## Fallback and retry strategies When the model call itself fails — not the tool execution, but the API call to the model provider — a fallback to a different model is the standard production response. The pattern is simple but the details matter. Same-family fallbacks (e.g., `claude-opus-4-5` → `claude-sonnet-4-5` on capacity errors) are safe by default because the tool schema format is identical. Cross-family fallbacks (e.g., Anthropic → OpenAI on sustained outage) require testing that your tool definitions render correctly in the target provider's format — the JSON Schema is the same, but parameter names like `tool_choice` differ between providers. ```python import anthropic from tenacity import retry, stop_after_attempt, wait_random_exponential, retry_if_exception_type PRIMARY_MODEL = "claude-sonnet-4-5" FALLBACK_MODEL = "claude-haiku-4-5" @retry( retry=retry_if_exception_type((anthropic.RateLimitError, anthropic.APIStatusError)), wait=wait_random_exponential(multiplier=1, max=30), stop=stop_after_attempt(3), reraise=True, # without this, tenacity raises RetryError and the fallback below never fires ) def call_model_with_retry(client, messages, tools, model=PRIMARY_MODEL): return client.messages.create( model=model, max_tokens=4096, tools=tools, messages=messages, ) def call_model_with_fallback(messages: list, tools: list) -> dict: client = anthropic.Anthropic() try: return call_model_with_retry(client, messages, tools, model=PRIMARY_MODEL) except (anthropic.RateLimitError, anthropic.APIStatusError) as e: # Log the primary failure before falling back log_model_fallback(primary=PRIMARY_MODEL, fallback=FALLBACK_MODEL, error=str(e)) return call_model_with_retry(client, messages, tools, model=FALLBACK_MODEL) ``` Exponential backoff with jitter is non-negotiable for retry logic — without jitter, a burst of failures triggers synchronized retries that hit the API at the same moment, causing the next wave of rate-limit errors. Use `wait_random_exponential`, not the deterministic `wait_exponential` — the plain variant produces exactly the synchronized herd you're trying to avoid. And note `reraise=True` in the decorator: tenacity's default behavior after exhausting retries is to raise its own `RetryError`, which would sail past the `except (RateLimitError, APIStatusError)` clause and skip the fallback entirely. Set a reasonable maximum retry count (3 is standard) and a maximum wait time (30 seconds is usually the right ceiling before you'd rather return an error to the user than keep them waiting). For tool execution failures, the retry decision should come from the structured error contract: `retryable: True` means try once more, `retryable: False` means surface the error immediately. ## Hallucinated tool calls A failure mode that doesn't get enough attention: the model invents a tool that doesn't exist in the provided tool list and tries to call it. This happens with larger tool sets (15+ tools) where the model may confuse tool names, with poorly named tools that sound similar to other tools, and when the model is trying to accomplish a task that genuinely requires a capability that isn't available. Don't reach for `tool_choice` here — it controls *whether* the model calls a tool, not *which names* it can emit: ```python # tool_choice "auto" is the default: the model decides whether to use a tool. # It does NOT constrain tool names — a confused model can still invent one. response = client.messages.create( model="claude-sonnet-4-5", tools=tools, tool_choice={"type": "auto"}, messages=messages, ) # tool_choice can require a *specific* tool on the first turn — useful for # forcing a known entry point, not for preventing hallucinated names later response = client.messages.create( model="claude-sonnet-4-5", tools=tools, tool_choice={"type": "tool", "name": "search_orders"}, messages=messages, ) ``` The real defenses are three. First, strict/constrained tool schemas where the provider guarantees the emitted name and arguments conform — the same constrained-decoding machinery covered in the [structured outputs article](/ai-engineering/articles/structured-outputs-constrained-decoding). Second, orchestrator-side name validation: the `execute_tool_with_validation` function earlier in this article already returns a structured `unknown_tool` error for names not in the registry, which gives the model something to recover from instead of an exception. Third, curation: keep the tool list small and the names unambiguous. When the tool list grows large (20+ tools), consider the Anthropic Tool Search Tool pattern: expose a `search_tools` meta-tool that returns relevant tool definitions on demand, so the active context only contains the 3–5 tools relevant to the current task. This also reduces the token overhead from tool definitions — Anthropic's system prompt injection for tool use runs 290–804 tokens depending on model and `tool_choice` setting, which multiplies across every iteration of a long loop. For parallel tool calls, OpenAI's strict mode (`strict: true`) is incompatible with `parallel_tool_calls`. If you need both schema guarantees and parallel execution, you must choose one or implement a two-pass approach: use non-strict mode for parallel calls, then validate outputs with Pydantic before passing results back to the model. See the [tool-use round-trip article](/ai-engineering/articles/tool-use-round-trip-agentic-loop) for the full mechanics of parallel call handling. ## What breaks (and when) ```mermaid flowchart LR subgraph "Failure modes by phase" A["Tool call generation\n(model → args)"] --> A1["Hallucinated tool name"] A --> A2["Schema-invalid args\n(if not using strict mode)"] A --> A3["Semantically wrong values\n(hallucinated IDs, wrong dates)"] B["Tool execution"] --> B1["Unhandled exception\n→ empty error return"] B --> B2["Duplicate execution\n(no idempotency key)"] B --> B3["Timeout with no result\n(model retries blindly)"] C["Loop management"] --> C1["No iteration cap\n→ runaway cost"] C --> C2["Context rot\n→ earlier results lost"] C --> C3["No budget accounting\n→ surprise bill"] D["Result handling"] --> D1["Unchecked refusal\n(OpenAI message.refusal)"] D --> D2["Parsing crash on\npartial JSON"] end style A1 fill:#ff2e88,color:#111 style A3 fill:#ffaa00,color:#111 style B1 fill:#ff2e88,color:#111 style B2 fill:#ff2e88,color:#111 style C1 fill:#ff2e88,color:#111 style C2 fill:#ff2e88,color:#111 style D1 fill:#ffaa00,color:#111 ``` **Context rot** deserves its own treatment because it's slow-moving and hard to notice. As a multi-step loop accumulates tool results, the context grows. Earlier tool results — the customer profile fetched in step 1, the account balance from step 2 — eventually fall outside the model's effective attention window even if they technically fit within the token limit. The model starts contradicting earlier steps, re-fetching data it already has, or generating inconsistent arguments. This is not a model bug — it's a consequence of how attention works over long sequences. See [context window management for agents](/ai-engineering/articles/context-window-management-agents) for the compress/summarize strategies. **The OpenAI refusal case** is a specific footgun worth naming: when OpenAI Structured Outputs declines to generate a response (due to content policy or model uncertainty), the response object contains a `message.refusal` field instead of a parsed result. Code that blindly calls `response.choices[0].message.parsed` without first checking `response.choices[0].message.refusal` will throw an `AttributeError` rather than handling the refusal gracefully. Always check `message.refusal` before accessing `message.parsed`. **Timeout with no result** is different from a clean error. If a tool call times out and you return `None` or an empty string, the model treats it as a successful tool call with no data and tries to reason from it. Return a structured timeout error with `error_type: "timeout"` and `retryable: True` — the model can then decide to try again or tell the user the operation is taking longer than expected. **Aborting mid-chain leaves partial state.** The budget-exceeded return from `run_agent` hands back `partial_messages` — but if the chain had already executed two of three write tools when the cap hit, the external world is now inconsistent: an order exists with no scheduled delivery, a charge posted with no confirmation record. Three mitigations, in order of preference. Structure workflows read-first-write-last so a single write commits at the end and an abort before it costs nothing. Where multiple writes are unavoidable, pair each write tool with a compensating tool (`cancel_order` for `create_order`, `refund_charge` for `charge_card`) and have the orchestrator — not the model — walk the completed writes in reverse on abort, saga-style. And route every `budget_exceeded` or hard-failure session whose message history contains a successful write to a reconciliation queue with human review; the idempotency store doubles as the audit trail, because its keys tell you exactly which writes committed. ## Observability: tracing what actually happened You cannot debug a multi-step tool-using system from model-level traces alone. You need a span per tool call that captures the input arguments, the raw output, the execution latency, and the token cost of the surrounding model call. The pattern is a decorator or context manager that wraps each tool execution: ```python import time from contextlib import contextmanager from dataclasses import dataclass, field from typing import Any @dataclass class ToolSpan: tool_name: str session_id: str iteration: int args: dict result: dict | None = None error: str | None = None latency_ms: float = 0.0 model_input_tokens: int = 0 model_output_tokens: int = 0 extra: dict = field(default_factory=dict) def trace_tool_call(span: ToolSpan, emit_fn): """Emit a span to your observability backend (Langfuse, Braintrust, custom).""" emit_fn({ "type": "tool_span", "tool_name": span.tool_name, "session_id": span.session_id, "iteration": span.iteration, "args_summary": summarize_args(span.args), # truncate PII before logging "result_status": span.result.get("status") if span.result else "error", "error_type": span.error, "latency_ms": span.latency_ms, "model_tokens": span.model_input_tokens + span.model_output_tokens, }) ``` The fields that matter most for failure localization: - `latency_ms` per tool call lets you find which tool is slowing down the loop. - `result_status` lets you see error rates per tool in aggregate. - `iteration` lets you see how far into the loop most sessions get before completing or failing. - `model_tokens` per iteration shows you context growth over the loop — the telltale sign of context rot is accelerating token counts in later iterations. Production agentic evaluation frameworks — including [TRAIL](https://arxiv.org/abs/2505.08638) and [AgentCompass](https://arxiv.org/abs/2509.14647) (both 2025) — emphasize per-tool-call tracing as the foundational observability layer for agentic issue localization. Without it, a five-step agent chain that fails on step four looks like a single failed request rather than a sequence of partially-successful steps with a recoverable failure at the end. ## The decision in practice There's a spectrum from "fully deterministic pipeline" to "fully autonomous agent" and the right point on it depends on your tolerance for failures and the complexity of the task. For structured data extraction (fill out a form, classify a document, extract entities), use a single model call with [strict-mode structured outputs](/ai-engineering/articles/structured-outputs-constrained-decoding) and a Pydantic validation layer. No loop needed — if the first attempt fails validation, retry once with the error appended. For single-domain tasks (answer a question about a customer's account by calling 2–3 read tools), a bounded loop with `max_iterations=5` is appropriate. The task space is narrow enough that 5 iterations should always be sufficient; if it isn't, you have a prompt problem, not a loop budget problem. Multi-step workflows (research a topic, draft a response, schedule a follow-up) call for `max_iterations=15–20` with careful context management. Use summarization to compress earlier steps and prevent context rot. Emit a span per step so you can see where sessions are spending time. Open-ended research or code generation tasks that may genuinely need 30+ steps should treat the loop as infrastructure requiring the same monitoring as any production service: alerting on runaway cost per session, dashboards showing P50/P95 iterations-to-completion, and a human-in-the-loop escalation path for sessions that hit the budget cap without converging. The [multi-agent orchestration article](/ai-engineering/articles/multi-agent-orchestration-patterns) covers the coordination patterns for tasks that are too complex for a single loop. [Interactive visualizer on the original page: agent-loop — Bounded agent loop with structured error handling] The instinct to build the fully autonomous agent first is understandable — it's the impressive demo. The production reality, confirmed repeatedly across industry analyses, is that systems built around deterministic orchestration with small, well-monitored agent sub-loops ship faster, fail more gracefully, and are far easier to debug when they do fail. Start there. ## Frequently asked questions Q: Why do tool-using LLM systems fail in production when they work fine in demos? A: Demo environments use happy-path tool inputs, short loops, and small context windows. Production breaks in four main ways: context rot (the conversation grows until earlier tool results disappear from the window), unbounded loops (no iteration cap means a confused model burns tokens forever), semantic errors (structured output is schema-valid but the field values are wrong), and missing error handling (tool execution failures return nothing useful, so the model hallucinates a retry). Adding explicit loop budgets, structured error returns, and a semantic validation layer after schema validation closes most of the gap. Q: How do I prevent runaway agentic loops from consuming unbounded tokens? A: Set an explicit iteration cap (3–10 steps for focused tasks, 20–30 for research-style workflows) and a cumulative token budget before the loop starts. Track both counts inside the loop and return a structured failure when either limit is reached — do not just raise an exception. DoorDash calls this "budgeting the loop" and treats it as the single most important reliability primitive in their agentic platform. A reasonable starting point: max_iterations=10, max_tokens=50000 for a single-task agent. Q: What is the difference between schema validation and semantic validation for tool calls? A: Schema validation checks structure: is the JSON syntactically valid, are required fields present, do values match the declared types? Semantic validation checks meaning: does this order_id actually exist in the database, is this date in the future, is this quantity a positive integer, does this user have permission for this action? Constrained decoding and strict-mode schemas handle the first problem. You must implement the second layer yourself, typically in a Pydantic or Zod validator that runs after parsing and before tool execution. A hallucinated order ID passes all schema checks and fails at the database, usually with an unhelpful exception. Q: Should I use strict mode (strict: true) for tool schemas in production? A: Yes, for most production use cases — but with two caveats. On OpenAI, strict: true requires additionalProperties: false and every field marked required, and it is mutually incompatible with parallel_tool_calls. On Anthropic, strict tool schemas (2025 beta) provide the same guarantee with similar constraints. Start with strict mode off for rapid iteration, lock it in before deployment. The performance overhead is negligible; the reliability gain from eliminating schema-invalid tool calls is substantial. Q: How should tool execution errors be returned to the model? A: Return a structured error object in the tool_result rather than raising an exception or returning an empty string. The object should include an error_type (e.g., "not_found", "permission_denied", "rate_limited"), a human-readable message, and whether the error is retryable. Models reason significantly better about next steps when the error is typed and explicit. An empty or cryptic error causes the model to either hallucinate a successful result or retry identically — neither is useful. Q: What does it mean to make a tool call idempotent, and why does it matter? A: An idempotent tool call produces the same result if executed once or multiple times with the same inputs. It matters because agentic loops retry on uncertainty, models sometimes produce duplicate parallel calls, and network failures can cause tool execution to happen while the result is lost. For read operations idempotency is free. For writes, implement it with idempotency keys: the caller generates a stable key per logical operation (e.g., hash of (session_id, tool_name, canonical_args)), and the tool implementation deduplicates on that key. This prevents double-charges, duplicate records, and conflicting state changes.