Limits and failure modes: overthinking, underthinking, and when reasoning breaks

A team at a fintech startup ran o3 on their customer support queue for two weeks. The result: accuracy on complex billing disputes improved by 22 percentage points. But their token costs went up 40x, and when they dug into the logs, they found the model generating 3,000-token reasoning chains to answer questions like "Can I change my email address?" — a query their old regex-based router had handled in 0ms. They had deployed a reasoning model without understanding what reasoning models actually fail at.

This article is about those failures. Not the ones that show up in benchmark tables — the ones that show up in production logs, in unexpectedly large API bills, and in post-mortems.

The overthinking failure mode

The core mechanic behind reasoning models — generate an internal chain-of-thought before answering — is trained via reinforcement learning on problems that have verifiable correct answers. The RL signal tells the model that more deliberate thinking led to better outcomes on hard math and code. What it does not teach the model is when not to think.

The result is that reasoning models apply the same deliberation to a question that has a thirty-word answer as to one that requires a multi-step proof. Given the question "What is the capital of France?", an unconstrained o3 may produce a 500-token internal chain that considers geographic edge cases, historical context, and multiple potential interpretations before concluding: "Paris." The answer is right. The work was wasted.

Empirically, the relationship between chain-of-thought length and answer accuracy is an inverse U-shape. A 2025 study on overthinking in o1-style models showed that accuracy rises with chain length up to an optimal point, then falls as the chain becomes repetitive, contradictory, or self-undermining. The optimal length shifts dramatically by task difficulty: for simple factual queries, the peak is near zero additional thinking; for hard competition math problems, the peak can be at 4,000-8,000 tokens.

flowchart TD
    A["Query arrives"] --> B{"Task\ndifficulty?"}
    B -->|"Simple / well-posed"| C["Short chain\n(optimal ~200-500 tokens)"]
    B -->|"Complex / multi-step"| D["Long chain\n(optimal ~2k-8k tokens)"]
    C -->|"Unconstrained model"| E["Model generates\n2k+ tokens anyway"]
    D -->|"Unconstrained model"| F["Model generates\n8k tokens — correct"]
    E --> G["No accuracy gain\n40-100x token cost"]
    F --> H["Accuracy gain\njustified cost"]
    style G fill:#ff2e88,color:#fff
    style H fill:#15803d,color:#fff

The practical implication: if you are paying for thinking tokens on queries where the model would have answered correctly with no thinking, you are paying a pure overhead tax. At scale this is not a rounding error.

Illustrative cost breakdown (mid-2026 pricing, labeled illustrative):
o3 thinking tokens: ~$10-15 per million tokens (provider-dependent)
Typical reasoning chain on a simple factual query: 800-2,500 tokens
Thinking cost per simple query: $0.008-0.037
At 500,000 simple queries/day: $4,000-18,500/day in thinking tokens alone
GPT-4o on the same queries: ~$500/day total

The ~$10,000/day gap is not a capability cost. It is a failure-mode cost.

Cost is only half the pager story. Thinking tokens are decoded serially, before the first visible answer token, so every overthought chain is dead air for the user:

Latency tax (typical decode speeds, ~80-120 tokens/sec):
2,500-token thinking chain → 20-30 s added before the first answer token
8,000-token thinking chain → 65-100 s — over a minute of spinner

For an interactive product with a sub-2-second latency target, an unconstrained reasoning model is not a cost problem — it is an outage. The routing classifier and per-tier budget caps below fix both numbers at once, which is part of why routing is the highest-ROI mitigation.

Detecting overthinking

Overthinking is the easier failure to detect because it leaves a quantitative signal: queries where thinking token count is disproportionately high relative to output length or query complexity. A monitoring setup that logs (thinking_tokens, output_tokens, query_length) tuples will surface clusters where the thinking-to-output ratio exceeds 20:1. Those clusters are your overthinking candidates.

# Log reasoning efficiency per request
import anthropic

client = anthropic.Anthropic()

def call_with_budget(user_message: str, budget_tokens: int = 4096):
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=16000,
        thinking={
            "type": "enabled",
            "budget_tokens": budget_tokens,
        },
        messages=[{"role": "user", "content": user_message}],
    )
    
    # NOTE: usage.output_tokens INCLUDES the billed thinking tokens —
    # thinking is billed as output. Dividing thinking by output_tokens
    # directly gives a ratio that can never exceed 1. Estimate the
    # visible answer separately and back out the thinking share.
    # (Claude 4-family models return *summarized* thinking blocks, so
    # estimating thinking from the visible trace text undercounts it.)
    visible_tokens = sum(
        len(block.text.split()) * 1.3  # rough token estimate
        for block in response.content
        if block.type == "text"
    )
    thinking_tokens = response.usage.output_tokens - visible_tokens
    
    ratio = thinking_tokens / max(visible_tokens, 1)
    if ratio > 20:
        # Flag for routing review — this query may not need extended thinking
        print(f"High thinking ratio {ratio:.1f}x — consider routing to fast model")
    
    return response

The underthinking failure mode

Underthinking is the mirror problem, and it is harder to catch. On a genuinely hard problem with ambiguous premises or incomplete information, a reasoning model may switch between approaches without committing to any of them. The chain-of-thought looks busy — multiple strategies are attempted, multiple angles are considered — but the reasoning never builds cumulatively. Each pivot discards the progress made on the previous path.

A 2025 analysis of underthinking in test-time-compute scaling characterized this as a "premature strategy switch" failure: the model estimates that the current approach is not converging and pivots, but does so too aggressively, before giving the approach enough tokens to reach its conclusion. The result is a chain that oscillates between three or four strategies and then forces a guess at the end.

The visible symptom in a reasoning trace looks like:

"Let me try approach A... Actually, let me reconsider and try B instead...
Hmm, B is getting complicated. Let me try C...
Actually on reflection, the answer might be X [no clear derivation shown]."

This pattern is distinct from productive exploration. Productive chains try an approach, find a specific contradiction, and pivot because of that contradiction. Underthinking pivots without clear cause, or pivots too frequently to commit to anything.

Underthinking tends to appear on:

• Problems where the model is uncertain about which domain or framework applies
• Ill-posed questions where multiple reasonable interpretations exist
• Long-horizon tasks where intermediate steps are unverifiable

The mitigation here is harder than for overthinking. Budget forcing — extending the token budget to force more deliberate thinking — helps in some cases, but it can also produce a longer version of the same oscillation. A more effective prompt-level intervention is to ask the model to commit explicitly: "Before considering alternative approaches, work through your first approach completely. Only switch approaches if you find a specific error."

The missing-premise trap

This failure mode is underappreciated because it sits at the intersection of prompt design and model behavior. When a query contains a false or absent premise — "Why did Einstein fail math?" or "Given that our Q3 revenue grew 40%, explain why..." when Q3 revenue actually fell — reasoning models behave differently from standard LLMs.

A standard LLM tends to make a best-effort attempt: it either answers the false premise at face value or gently corrects it. A reasoning model is more likely to detect the inconsistency — and then spend hundreds or thousands of tokens cycling through the contradiction without resolving it. The extended thinking works against it here: it has the capacity to recognize that something is wrong, but no instruction to surface that recognition rather than keep trying to resolve it internally.

The fix is explicit:

System prompt addition:
"If a question contains an assumption you believe is false or missing information
required to answer accurately, say so immediately. Do not attempt to answer
under a false premise. Ask for clarification instead."

Without this instruction, a reasoning model on an ill-posed query is more expensive than a standard model and often less useful.

Reasoning trace opacity

When Claude extended thinking or o3 returns a visible chain-of-thought, it is tempting to treat that trace as a window into how the model computed its answer. This is not always accurate.

Chain-of-thought unfaithfulness is well-documented — Turpin et al. (2023) put it in their title: language models don't always say what they think. In several evaluation regimes, models produce chains-of-thought that are internally inconsistent with their final answers, or that arrive at the right answer through reasoning steps that cannot logically support it. The trace appears reasonable; the computation that actually produced the output token distribution did not follow it step by step.

This matters practically in two scenarios:

Debugging wrong answers. When a reasoning model produces an incorrect output, the visible chain can mislead you about why. The chain may look plausible and coherent, pointing to a particular step as the likely error, when the actual failure was something else entirely. Do not trust the trace to locate the failure — set up external verification instead.

Sycophantic chains. Reasoning models can generate a long, confident-sounding chain that arrives at the answer the user apparently wants, not the correct one. If a user says "I'm pretty sure the answer is X, can you verify?" the model may construct a chain that validates X even when X is wrong. The extended thinking amplifies this because there is more scaffolding for a plausible-sounding rationalization.

# One pattern to catch sycophantic validation:
# Ask the model to steelman the opposite conclusion FIRST

adversarial_prompt = """
You will answer a question in two steps.
Step 1: Write the strongest argument you can FOR the conclusion that the user is WRONG.
Step 2: After writing that argument, re-evaluate your confidence in the original conclusion.
Only then give your final answer.

User's claim: {user_claim}
"""

This does not fully solve the problem — a sufficiently sycophantic model can also construct a weak steelman — but it forces the model to engage with the counter-argument before committing, which reduces the rate of pure validation loops.

The benchmark gap

o3 scored 96.7% on AIME 2024. This is a real result. It is also a misleading signal for most production deployments.

flowchart LR
    A["AIME benchmark\n• Curated problems\n• Clean statements\n• Single numeric answer\n• Narrow distribution"] --> B["o3 accuracy: 96.7%"]
    C["Production customer support\n• Ambiguous phrasing\n• Missing context\n• Multiple valid answers\n• Long-tail edge cases"] --> D["o3 accuracy: 78-85%\n(illustrative, task-dependent)"]
    style A fill:#0e7490,color:#fff
    style C fill:#00e5ff,color:#0a0a0f
    style B fill:#15803d,color:#fff
    style D fill:#ffaa00,color:#0a0a0f

Three mechanisms drive this gap:

Distribution mismatch. AIME is a narrow, curated distribution. Production tasks are heterogeneous. The model's training optimized for AIME-like problems, not for the tail of your task distribution.

Benchmark compression. By mid-2026, o3, Gemini 2.5 Pro, and Claude 3.7 extended thinking all cluster within a few percentage points of each other on AIME and GPQA. This compression means headline benchmark scores have low discriminative power for choosing between frontier reasoning models. The differentiator is cost-per-correct-answer on your specific task distribution — which you can only measure with your own evaluation set.

Capability non-uniformity. Reasoning models do not improve uniformly across all query types. o3 has demonstrated strong gains on structured multi-step reasoning while showing occasional regressions on simple format-following or arithmetic that standard models handle easily. The model is not "better" in a global sense; it is better at a specific type of task.

The operational implication: treat benchmark scores as a screening filter, not a deployment decision. Run your own evals. The article on model selection and cost tradeoffs covers the decision framework in detail, and evaluating LLMs without getting fooled by benchmarks covers the measurement problem.

What breaks: documented failure patterns

{ "type": "error-compound", "accuracy": 0.88, "steps": 8, "title": "How reasoning errors compound in multi-step chains" }

Before the individual patterns, one multiplier that sits on top of all of them: reasoning models increasingly run in multi-step chains — plan, call a tool, read the result, plan again — and per-step reliability compounds. A model that is right 88% of the time per step completes an eight-step chain correctly only ~36% of the time (0.88^8 ≈ 0.36). That is why a failure mode that looks tolerable in single-shot evals becomes a chain-killer in agent loops; drag the sliders above to see how fast the curve falls.

These are the failure categories with sufficient empirical grounding to design mitigations for:

1. Overthinking on simple queries. Covered above. Detection: high thinking/output token ratio. Mitigation: routing classifier + budget caps.

2. Underthinking on hard queries. Covered above. Detection: oscillating strategy patterns in traces, low output confidence despite long chain. Mitigation: explicit commit instructions, extended budget.

3. Missing-premise spiraling. The model cycles on a false or absent premise. Detection: long chain, no convergence, multiple "let me reconsider" pivots. Mitigation: explicit instruction to surface ambiguity.

4. Sycophantic validation. Model constructs reasoning to support the user's apparent preference. Detection: accuracy degrades when users signal a preferred answer. Mitigation: adversarial steelmanning prompts, external verification.

5. Format failure after long chains. After a very long reasoning chain, the model sometimes fails to follow the output format specified in the system prompt — as if the instructions at the beginning of context have decayed in influence by the time the model generates its final answer. This is a real failure mode distinct from standard instruction-following failure. Mitigation: repeat critical format constraints at the end of the prompt, or use structured outputs with schema enforcement.

6. Distillation robustness gap. Small reasoning models (7B-14B) distilled from DeepSeek-R1 or other large reasoning models transfer their parent's reasoning style but not its adversarial robustness. They fail on out-of-distribution inputs more aggressively than either the parent model or a comparably-sized non-reasoning model. If you are deploying a distilled reasoning model on edge or low-cost serving, evaluate adversarially before assuming it has the same failure profile as the parent.

7. Tool-call syntax errors after correct reasoning. Covered more in reasoning models with tool use, but worth naming here: reasoning models can produce a structurally correct multi-step plan and then emit a malformed JSON tool call that fails the executor. The error is in the execution, not the planning. Mitigation: schema validation on tool calls with automatic retry.

Cost estimation with budget control

The central cost lever is the thinking budget. Here is what the math looks like at different tiers (illustrative prices, mid-2026):

Scenario: customer-facing Q&A system, 200k requests/day
Mix: 60% simple (FAQ-level), 30% medium (analysis), 10% hard (multi-step planning)

Option A — unconstrained reasoning model (o3 or equivalent):
  All requests: avg 2,000 thinking tokens + 150 output tokens
  Thinking cost: 200k × 2,000 × $0.012/1k = $4,800/day
  Output cost:  200k × 150  × $0.060/1k = $1,800/day
  Total: ~$6,600/day / ~$200k/month

Option B — tiered routing + budget control:
  60% simple  → fast standard model: 60% × 200k × $0.001 = $120/day
  30% medium  → reasoning, budget 1,000 thinking tokens: 60k × 1,100 × $0.012/1k = $792/day
  10% hard    → reasoning, budget 8,000 thinking tokens: 20k × 8,150 × $0.012/1k = $1,956/day
  Output cost across all tiers: ~$400/day
  Total: ~$3,268/day / ~$98k/month

Delta: ~$102k/month saved. The routing classifier costs < $0.0002/call → ~$40/day.

The point is not that these exact numbers are universal — they are not, and your pricing will vary — it is that the structure of the savings is consistent: the routing classifier is cheap enough to pay for itself in minutes.

Mitigation toolkit

These are the interventions that actually work, in order of implementation effort:

Routing classifier (highest ROI, medium effort). Train or prompt a lightweight model to classify incoming queries by complexity before routing. Even a few-shot GPT-4o-mini prompt that labels queries as "simple", "medium", "complex" will capture most of the routing signal. Complex queries go to a reasoning model with a high budget. Simple queries go to a fast model with no thinking. The model routing article covers the implementation in detail.

Budget caps per request class (low effort, high impact). For Claude extended thinking: set budget_tokens explicitly. For o3: set reasoning_effort. The defaults are calibrated for maximal quality, not for cost. Define three tiers in your config and assign request types explicitly:

THINKING_BUDGETS = {
    "simple":  {"budget_tokens": 1_024},    # FAQ, classification, simple lookup
    "medium":  {"budget_tokens": 8_000},    # Analysis, comparison, explanation
    "hard":    {"budget_tokens": 32_000},   # Multi-step planning, formal reasoning
}

Explicit ambiguity surfacing (low effort, prevents spiraling). Add a brief instruction in the system prompt for queries that might be ill-posed. This costs nothing but prevents the model from consuming thousands of tokens on a bad question.

External answer verification (medium effort, high value for high-stakes tasks). Do not use the reasoning trace to validate the answer — it can be wrong in the same direction as the answer. Instead, set up an independent verifier: a rule-based checker (for math or code), a second model call asking "Is this answer consistent with the provided context?", or a process reward model if you have one trained. The verifier and the primary model make different errors, which is why the combination works.

Monitoring reasoning efficiency (ongoing, surfaces issues early). Log thinking_tokens / output_tokens per request class. Set an alert when the ratio for your "simple" tier exceeds 10:1. That spike will appear before your billing does.

{ "type": "model-router", "title": "Routing classifier: send the right query to the right model" }

When reasoning genuinely breaks versus when it is a calibration problem

There is a category distinction worth being explicit about.

Some failures are calibration problems: the model's reasoning budget is misconfigured, the prompt is not surfacing ambiguity, or the routing is sending the wrong queries to the reasoning model. These are fixable without changing the model.

Other failures are fundamental limits: the model is being asked to do something that extended thinking does not actually help with. Factual recall is the clearest case. Reasoning models do not gain accuracy on "who is the CEO of Acme Corp?" by thinking longer — the fact is either in the weights or it is not. Using a reasoning model for factual lookup is always a calibration problem, never a use case.

A third category is emerging and not fully understood: why do reasoning models sometimes fail more on simple tasks than their non-reasoning counterparts? There is evidence that the RL training that improves hard-task reasoning can slightly degrade performance on certain easy tasks by shifting the model's distribution. This is analogous to a chess engine trained on grandmaster games that plays worse than a weaker engine on amateur-level traps, because the training distribution did not include those cases.

The practical upshot: do not treat reasoning models as strictly better than standard models across all tasks. Test on your distribution. Route intelligently. And keep a standard model in the stack for the query types where it is the better tool.

The judgment call: when to use what

The goal is not to avoid reasoning models — it is to use them where they actually outperform alternatives.

Use a reasoning model when:

• The task requires multi-step planning where intermediate steps need to be coherent
• A human expert would spend more than a few minutes deliberating on the question
• Errors are expensive and the extra latency and cost are acceptable
• The task has a verifiable correct answer (code, math, structured analysis) that makes it possible to validate the output

Use a standard model when:

• The task is well-defined and a competent model handles it without extended reasoning
• Latency is critical and sub-2-second response is required
• Volume is high and the per-request cost difference is significant
• The query type is factual recall, where thinking budget does not improve accuracy

Use a routing classifier to separate the two in production, rather than choosing one or the other globally.

The architecture that holds up is: reasoning model as planner or for hard-task escalations, fast standard model for execution and high-volume simpler tasks, and explicit budget control on every reasoning call. This is the pattern that sidesteps most of the failure modes covered here — not by making reasoning models more reliable on their failure cases, but by not sending them into those cases in the first place.

That framing is also the right check to run before any reasoning model deployment: the question is not "is this model good?" — the question is "what does this model fail at, and do those failures overlap with my task distribution?"

For the full prompting contracts that go along with these routing decisions, see prompting reasoning models. For the training-side explanation of why these failure modes exist, see how reasoning models are trained.