Agent Evaluation and Why Agents Fail in Production

Your agent demo went perfectly. The model browsed the web, called three APIs, compiled a report, and emailed it — all in forty seconds. Three weeks later a user reports that the report contained a confident citation to a document that does not exist, the email went to the wrong address because a tool returned an ambiguous result and the agent guessed, and the entire loop had silently re-run twice on the same task because the completion signal was not deterministic. Nothing crashed. The logs showed status: success. That is the failure mode that kills production agents.

The math that explains 85% of pilot failures

Start with the number: fewer than 15% of agent pilots reach production scale. That is not a product-market-fit problem or a compute problem. It is an evaluation and reliability problem, and the math underneath it is not mysterious.

If an agent executes a multi-step workflow where each step succeeds with probability p, the probability of the entire workflow succeeding is p^n. This is the compounding error formula, and it is ruthless:

p = 0.85, n = 10  →  0.85^10 ≈ 0.20   (20% end-to-end success)
p = 0.90, n = 10  →  0.90^10 ≈ 0.35   (35% end-to-end success)
p = 0.95, n = 10  →  0.95^10 ≈ 0.60   (60% end-to-end success)
p = 0.99, n = 10  →  0.99^10 ≈ 0.90   (90% end-to-end success)

Treat p^n as the open-loop floor, not destiny: it assumes steps are independent and unrecoverable. Retries and structured error recovery (covered below) raise effective per-step accuracy above the raw number, while correlated failures — one bad tool result poisoning the context for every later step — can push you below it.

Getting to 90% per-step accuracy — where 10-step tasks succeed 35% of the time — is genuinely hard. Current production agents routinely operate at 80–90% per-step accuracy on well-scoped tasks. That means most multi-step agentic workflows fail more often than they succeed.

The industry's benchmark problem amplifies this. Evaluation sets used in labs are curated: known inputs, clean tool responses, one plausible path to the answer. Production traffic has ambiguous requests, tools that return unexpected formats, rate-limited APIs that return partial results, and context windows that fill up mid-task. The gap between lab benchmarks and production deployment is consistently large — practitioners report 30-40 percentage points in accuracy drop across task domains. A system hitting 90% on a curated leaderboard commonly delivers 60-70% reliability in the wild.

{ "type": "error-compound", "accuracy": 0.85, "steps": 10, "title": "Compounding error: why per-step accuracy lies" }

What actually breaks (taxonomy of production failures)

Tool misuse is the dominant proximate cause — estimates from production failure analyses such as Arize's agent failure taxonomy place it at roughly 30% of failures, though the exact split varies by domain. It breaks into three distinct sub-problems. The first is wrong parameters: the model calls the right tool but passes a malformed argument — a string where an integer is required, a relative date string where ISO 8601 was specified, a customer ID from the wrong namespace. The second is wrong tool selection: given twenty tools in context, the model picks one that sounds right but has subtly different semantics. The third is hallucinated tool calls: the model invents a tool name that does not exist in the registered schema, then either errors out or — worse — the orchestration layer silently drops the call and the agent proceeds as if it succeeded.

The second major cluster is context degradation, which manifests in several ways. Over a long multi-turn conversation, the agent's effective system prompt gets diluted by the growing message history. Earlier constraints ("always cite sources", "never modify records without confirmation") become statistically weaker relative to the total token count and start failing. The agent also forgets what steps it has already completed and repeats them, or loses track of which sub-task it is currently executing. This is sometimes called "context rot" — the conversation accumulates enough noise that reasoning quality drops even though no individual message is wrong.

Silent wrong answers are the hardest failure mode to detect because there is nothing to catch. No exception, no log anomaly, no status: failure. The agent completes its loop, appends a plausible-looking result to the conversation, and the host application forwards it to the user. A medical information agent gives a confident but incorrect dosage. A financial agent writes a summary that inverts the sign on a return figure. A code agent generates code that compiles but has off-by-one logic. Each of these would fail a careful human review; none of them fail automated checks that only look for structural correctness.

Termination failures round out the list. An agent that has no explicit completion criteria relies on self-assessment: the model decides when it is done. Models are optimistic about their own completion status, which leads to either premature termination ("I believe I have completed the task" after step 3 of 8) or infinite loops where the agent cycles through the same tool calls because the expected termination signal never arrives. See the agent loop mechanics article for the hard-stop patterns that address this.

One class deliberately gets a pointer rather than a full treatment here: adversarial inputs. Prompt injection — direct, or smuggled in through tool outputs and retrieved documents — is a distinct failure category, not a variant of tool misuse: the agent does exactly what its context tells it to, and the context has been poisoned by an attacker. Every evaluation layer in this article assumes honest inputs, so a team can ship the full stack below and still be blind to it. The indirect prompt injection article covers the attack surface and the (incomplete) defenses.

The percentages in the taxonomy below are order-of-magnitude estimates from that production data, not precise measurements — expect your own split to differ.

flowchart TD
    F["Production failure"] --> T["Tool misuse ~30%"]
    F --> C["Context degradation ~24%"]
    F --> S["Silent wrong answers ~21%"]
    F --> TM["Termination failures ~14%"]
    F --> O["Other ~10%\n(incl. adversarial inputs)"]

    T --> T1["Wrong parameters"]
    T --> T2["Wrong tool selected"]
    T --> T3["Hallucinated tool name"]
    C --> C1["Constraint drift"]
    C --> C2["State repetition"]
    C --> C3["Context window exhaustion"]
    S --> S1["Structurally valid but wrong"]
    S --> S2["Confident hallucination"]
    TM --> TM1["Premature exit"]
    TM --> TM2["Infinite loop"]

    style F fill:#ff2e88,color:#111
    style T fill:#00e5ff,color:#0a0a0f
    style C fill:#ffaa00,color:#0a0a0f
    style S fill:#a855f7,color:#fff
    style TM fill:#0e7490,color:#fff

Why LLM-as-judge is not enough

LLM-as-judge is appealing: it is cheap to run at scale, it handles open-ended outputs that rule-based checks cannot, and for many tasks it aligns with human ratings acceptably well. For agent evaluation specifically, it has documented failure modes that make it unreliable as a sole evaluator.

Position bias is the most studied: when a judge is shown two responses in a pairwise comparison, it systematically prefers one position (usually first) independent of quality. The effect is large enough that reversing the order of candidates can flip the verdict. Studies find that LLM judges exhibit position bias in more than 50% of trials where the true quality difference is small.

Length bias compounds this. LLM judges tend to prefer longer, more elaborate answers. An agent that produces a verbose but wrong explanation of its reasoning scores higher than one that gives a terse but correct answer. For tool-using agents, this is particularly misleading: a model that writes three paragraphs of justification for a wrong tool call looks better than one that calls the right tool with no explanation.

Agreeableness bias is subtler: when the judge has seen the same model produce a previous answer, it tends to rate the follow-up more favorably regardless of content. In a multi-turn agent evaluation, a judge that has rated steps 1-5 positively is primed to rate step 6 positively even if step 6 is where the agent went off the rails.

The calibration problem is that you cannot know how biased your judge is without running calibration experiments: swap positions, vary lengths artificially, and score against human labels. At that point you have already spent most of the savings that made LLM-as-judge attractive.

The right role for LLM-as-judge is as one signal in a multi-layer pipeline — useful for flagging obvious quality failures in production traffic, useless as the final word on whether a trajectory was correct. See the detailed guide to LLM judge calibration for the calibration protocol.

The six evaluation layers

Production agent evaluation requires a stack, not a single test. Each layer catches different failure modes; none is sufficient alone.

Layer 1: Outcome evaluation on a golden test set. The foundation. For each test case, the agent runs end-to-end and the final output is compared against a known correct answer. Golden sets should include at least one representative from each known failure category and should be seeded with real production traffic samples. The tooling — Braintrust, Ragas, DeepEval — differs mainly in how they handle multi-step outputs and LLM-scored rubrics. The golden set is your regression guard, not a performance ceiling: it tells you when something has gotten worse, not whether the current level is good enough.

Layer 2: Trajectory evaluation. The agent may reach the correct answer by a bad path, or fail to reach the correct answer despite individually-correct steps. Trajectory evaluation audits the step-by-step execution: which tools were called, in which order, with which arguments, and whether any steps were repeated, skipped, or taken with wrong parameters. This requires recording the full execution trace for each test run. LangGraph and the OpenAI Agents SDK both emit structured traces; the agent trajectories evaluation article covers the schema in detail.

A concrete trajectory check:

from dataclasses import dataclass
from typing import Sequence

@dataclass
class ToolCall:
    tool_name: str
    arguments: dict
    result: dict | None
    error: str | None

def audit_trajectory(
    trace: Sequence[ToolCall],
    expected_tools: Sequence[str],
    registered_tools: Sequence[str],
    forbidden_tools: Sequence[str] = (),
) -> dict:
    called_names = [tc.tool_name for tc in trace]
    registered = set(registered_tools)
    expected = set(expected_tools)
    forbidden = set(forbidden_tools)
    # Hallucinated = not in the registered schema at all.
    # Unexpected = real tool, just not one this test case should need.
    hallucinated = [n for n in called_names if n not in registered]
    unexpected = [n for n in called_names if n in registered and n not in expected]
    forbidden_used = [n for n in called_names if n in forbidden]
    error_steps = [tc for tc in trace if tc.error is not None]
    repeated_steps = [n for n in set(called_names) if called_names.count(n) > 1]

    return {
        "hallucinated_tools": hallucinated,
        "unexpected_tools": unexpected,
        "forbidden_tools_used": forbidden_used,
        "steps_with_errors": len(error_steps),
        "repeated_tools": repeated_steps,
        "total_steps": len(trace),
    }

Layer 3: CI regression suite. Every prompt change, model version update, or tool schema modification runs the golden set automatically. The regression signal is the delta: did win rate drop? Did a specific failure category increase? Without CI, a prompt edit that improves average quality can silently regress a critical edge case. Prompt versioning as code covers the CI integration pattern.

Layer 4: Shadow evaluation on production samples. A sample of real production traffic (typically 1–5%, depending on volume and cost tolerance) is routed to a secondary evaluation pipeline. The agent's response is sent to users normally; the evaluation pipeline independently scores the response — using LLM-as-judge for high-volume traffic, human raters for a smaller sub-sample. This is where you find failure modes that your golden set never anticipated, because real users do things you did not script.

Shadow evaluation has a cost structure that matters:

Illustrative monthly cost for shadow eval (1M production requests/month):
  5% sample = 50,000 evaluation runs
  Each eval run: ~1,500 tokens for judge prompt + response
  At $3/M input tokens (GPT-4o-class judge, illustrative):
    50,000 × 1,500 tokens = 75M tokens × $3/M = $225/month

  Verdict: affordable at 1M req/month; run it on 100M req/month and you are
  looking at $22,500/month for the eval alone — at which point lighter
  per-turn classifiers become cost-controlling.

Layer 5: Per-turn online classifiers. A lightweight classifier (fine-tuned small model or embedding-based classifier) runs on each agent turn in production, scoring for anomalies — unexpected tool selections, parameter formats that look malformed, outputs that contradict earlier context. The latency budget is strict: these classifiers must complete in under 90ms to avoid adding perceptible latency to interactive agents. They do not block the response; they emit anomaly signals that the observability layer aggregates.

Layer 6: Runtime guardrails. The final line. Input and output filtering for scope violations, PII, and unsafe content, running in parallel with the agent execution path. The guardrails article covers the implementation stack (NeMo Guardrails, Llama Guard, custom classifiers). Guardrails catch what all previous layers missed but only for detectable violations — they do not help with a factually wrong answer that is perfectly safe.

sequenceDiagram
    participant U as User request
    participant G as Guardrail (input)
    participant A as Agent loop
    participant T as Tool execution
    participant C as Per-turn classifier
    participant E as Outcome eval (async)
    participant GO as Guardrail (output)

    U->>G: incoming request
    G->>A: passes / blocks
    loop agent steps
        A->>T: tool call
        T-->>A: tool result
        A->>C: turn emitted (async, <90ms)
        C-->>A: anomaly signal
    end
    A->>GO: final response
    GO->>U: response / block
    A->>E: trace logged (async)
    E-->>A: eval scores (next cycle)

The interactive below compresses this stack into five stages: outcome and trajectory evals (layers 1–2) together form its unit-eval stage, shadow eval and per-turn classifiers (layers 4–5) fold into production sampling, and it adds one stage this article has only pointed at — pre-release adversarial testing, deliberately attacking the agent before launch, which the injection and guardrails articles linked above cover in full.

{ "type": "eval-pipeline", "title": "Agent eval safety net: stages and what each catches" }

Designing agents to fail gracefully

Evaluation catches failures after the fact. Architecture choices determine whether failures are catastrophic or recoverable.

Cap loop depth explicitly. Every agent framework — LangGraph, the OpenAI Agents SDK, any custom orchestrator — should have a hard maximum iteration count set by the calling code, not by the agent's self-assessment. Ten steps is a reasonable upper bound for fully autonomous loops; anything longer should checkpoint with a human or emit a structured partial result. The multi-agent orchestration patterns article covers human-in-the-loop checkpoints for irreversible actions.

Make tool errors structured. When a tool fails, the error message returned to the agent determines whether it can recover or spirals into a retry loop. An error that says HTTP 429 gives the agent nothing to work with. An error that says {"error": "rate_limited", "retry_after_seconds": 5, "quota_type": "per_minute"} gives it everything it needs to reason about next steps. The agent will use error messages as context — design them as prompts.

# Tool that returns structured errors the agent can reason about
def search_customer_records(customer_id: str) -> dict:
    try:
        result = db.query(customer_id)
        return {"status": "found", "record": result}
    except RateLimitError as e:
        return {
            "status": "error",
            "error_type": "rate_limited",
            "retry_after_seconds": e.retry_after,
            "suggestion": "Wait and retry, or use cached_customer_records for recent queries"
        }
    except NotFoundError:
        return {
            "status": "error",
            "error_type": "not_found",
            "suggestion": "Try fuzzy_search_customers with partial name instead"
        }

Keep tool count low. Tool hallucination rates increase with the number of tools in context. An agent given fifty tools in its context window will misselect tools at much higher rates than one given eight. If your use case genuinely requires many tools, implement dynamic tool loading — the agent discovers available tools from a catalog and loads specific tools into context on demand — rather than injecting all tools upfront. Anthropic's Tool Search feature (2025) is designed exactly for this pattern. As a rough heuristic: above fifteen tools in a single context, measure hallucination rate before assuming the agent will route correctly.

Log everything, structurally. An agent that fails silently leaves no trace to debug. Every tool call, every tool response, every model output, every stop reason, and every iteration count should be emitted as a structured trace event. The observability stack — LangSmith, Langfuse, Arize Phoenix, Braintrust — ingests these traces and provides the dashboards that let you distinguish a systematic tool-schema bug from a one-off model fluke. Without structured traces, your shadow eval pipeline has nothing to score. See tracing and observability tooling compared for the choice matrix.

The silent wrong answer problem

Every other failure mode in this article leaves a signal: an exception, an error response, a tool retry, a terminated loop. Silent wrong answers leave nothing. The agent executes completely, returns a plausible-sounding result, and the error only surfaces when a human or downstream system checks the output against reality.

Mitigating silent wrong answers requires a combination of four approaches. None is complete on its own.

Self-consistency sampling. Run the same agent task multiple times with temperature > 0 and check for agreement. If three independent runs give three different answers, that is a signal the task is under-constrained or the agent is guessing. For high-stakes tasks, treat disagreement as a failure requiring human review. Cost is a real constraint: running three independent agent traces on a long workflow multiplies your token bill by three.

Grounded output validation. For structured outputs, validate not just against the JSON schema but against business rules. A valid JSON object with a future refund_date before the purchase_date is schema-valid and semantically wrong. Build validation layers that check referential integrity, range constraints, and consistency between fields. Pydantic validators in Python handle most of this cleanly:

from pydantic import BaseModel, model_validator
from datetime import date

class RefundRecord(BaseModel):
    customer_id: str
    purchase_date: date
    refund_date: date
    amount_usd: float

    @model_validator(mode='after')
    def refund_after_purchase(self) -> 'RefundRecord':
        if self.refund_date < self.purchase_date:
            raise ValueError(
                f"refund_date {self.refund_date} cannot precede "
                f"purchase_date {self.purchase_date}"
            )
        if self.amount_usd <= 0:
            raise ValueError(f"amount_usd must be positive, got {self.amount_usd}")
        return self

Confidence elicitation. Ask the model to explicitly rate its confidence and list what information it is uncertain about. This does not prevent hallucination, but it surfaces cases where the model is extrapolating from weak evidence — and those cases deserve a flag, a fallback, or a human review trigger. The key is to elicit uncertainty before the final answer, not after: post-hoc confidence is much less calibrated.

Retrieval grounding. For fact-based agent tasks, require that each factual claim in the output be grounded in a retrieved source, and check that the source actually supports the claim. This is where agentic RAG loops earn their cost: the verification step catches a significant fraction of confident hallucinations that pure generation misses.

What breaks in multi-agent systems

Multi-agent systems amplify every failure mode above. The handoff — one agent passing context to another — is the reliability bottleneck: practitioners consistently report handoff context loss as the single most common cause of multi-agent pilot failures.

The problem is that agent B receives a summary or a structured payload from agent A, not the full conversation history that informed A's decisions. Implicit assumptions that agent A made — "the user confirmed they want the bulk pricing tier", "this query was already validated against the schema" — may not be in the handoff payload, so agent B operates on incomplete context and makes different assumptions.

Design handoffs as contracts, not as free-form messages:

from pydantic import BaseModel
from typing import Literal

class ResearchHandoff(BaseModel):
    """Structured handoff from research agent to synthesis agent."""
    task_id: str
    original_query: str
    retrieved_sources: list[dict]       # each: {url, title, excerpt, confidence}
    key_facts: list[str]                # agent A's extracted facts
    open_questions: list[str]           # things agent A could not resolve
    constraints: dict                   # e.g., {"max_length": 500, "tone": "formal"}
    completed_steps: list[str]          # what has already been done — do not repeat
    handoff_reason: Literal["complete", "specialized_skill_needed", "human_review"]

An explicit completed_steps field is particularly important: without it, agent B will often re-execute steps that agent A already ran, doubling cost and sometimes producing conflicting results.

For a full treatment of orchestration topologies and when each pattern breaks, see multi-agent orchestration patterns. For the context management problem specifically, context window management for agents covers the compression and isolation strategies.

Evaluation tooling (as of mid-2026)

The evaluation tooling market has consolidated around a few meaningful categories. For offline golden-set evaluation, Braintrust and DeepEval are the most feature-complete, with Braintrust better integrated into the OpenAI and Anthropic SDK patterns and DeepEval more popular in the open-source community. Ragas is purpose-built for RAG agents; its task-decomposition metrics translate reasonably well to tool-using agents.

For tracing and online observability, LangSmith has the deepest integration with LangGraph agents (expected, since they are from the same company). Langfuse is open-source and cloud-hostable, which matters for organizations with data-residency requirements. Arize Phoenix is strongest on the debugging side — root-cause localization across traces — rather than aggregate dashboards.

Neither category is a replacement for the other. You need both: offline evals catch regressions before release, online observability tells you what is actually happening in production with real users.

Tool	Primary use	Strength	Weakness
Braintrust	Offline eval, LLM-judge scoring	OpenAI/Anthropic SDK integration	Less strong on trace visualization
DeepEval	Offline eval, open-source	Community metrics library, extensible	Requires more setup for custom metrics
Ragas	RAG + agentic RAG eval	Task-decomposition metrics	Limited to retrieval-augmented patterns
LangSmith	Tracing + eval	LangGraph native, rich trace viewer	Tied to LangChain ecosystem
Langfuse	Tracing + eval	Open-source, self-hostable	Fewer built-in agent-specific metrics
Arize Phoenix	Production observability	Root-cause localization, drift detection	Eval dataset management is weaker

The decision: when to add evaluation complexity

Starting with a full six-layer evaluation stack on day one is wrong. The first agentic MVP should have exactly two things: an outcome eval on a golden set of 50–100 examples, and structured logging of every tool call. That combination tells you whether the agent is working at all and gives you the trace data to diagnose failures.

Add trajectory evaluation when you have identified a pattern of failures where the final outcome looks wrong but you cannot tell which step went wrong. Add CI regression when you have had at least one prompt or schema change that regressed a previously-passing case. Add shadow production eval when you have enough production traffic to sample meaningfully. Add per-turn classifiers when you have a high-volume production agent where LLM-judge costs become prohibitive.

The one thing you should never skip, regardless of stage: hard iteration caps with explicit token budgets. An agent that can loop forever will, eventually, loop forever — and it will do so at the worst possible moment, on a production request, at maximum cost. Cap it at construction time, not at post-incident review.

The 85% of pilots that fail are not failing because the technology does not work. They are failing because teams shipped without measuring — trusted the demo, skipped the eval, and found out in production that the math had been working against them from the start.

Further reading

• The Agent Loop: How LLMs Become Actors — the perceive→reason→act→observe cycle and hard-stop guardrails.
• Agent Evaluation: Trajectories, Tool Calls, and Multi-Step Correctness — deep dive on the trajectory eval schema and tooling.
• LLM-as-Judge: Calibrating Your Automated Evaluator — position bias, calibration protocol, and when to trust the judge.
• Multi-Agent Orchestration: Patterns That Survive Production — handoff contracts, human-in-the-loop, and topology selection.
• Context Window Management for Agents — compression and isolation strategies for long-running agent sessions.
• Designing Tool Schemas That Models Actually Understand — schema design patterns that reduce tool misuse rates.
• Anthropic Engineering, "Building Effective Agents" — the canonical "start simple" guidance, with honest assessment of multi-agent complexity costs.
• Arize AI, "Common AI Agent Failures" — production failure taxonomy with frequency data.