Reasoning Models: When and How to Use Them

A startup's engineering team ships a support chatbot, and it's good enough. Then the model vendor announces a reasoning model that obliterates every benchmark in sight. The team swaps it in across the board. Two weeks later, the bill has tripled, p99 latency has jumped from 800ms to 28 seconds, and accuracy on routine ticket classification has actually dropped. They file a post-mortem and blame the model.

The model didn't fail. The team sent the wrong jobs to it.

Reasoning models are a new class of tool with a specific performance envelope. They are extraordinary at problems that require chained logic — where getting step 3 wrong means steps 4 through 10 are garbage. They are wasteful and sometimes worse than standard models on tasks that don't need deliberation. This module covers the mechanism, the decision criteria, the prompting changes, the cost math, and the failure modes that aren't obvious until you've already paid for them.

What test-time compute actually is

Standard LLMs predict the next token. They do this fast, in a single forward pass per token, using patterns baked in during training. There's no reconsideration. Whatever reasoning happens is implicit in the weights.

Reasoning models are different. They're trained with reinforcement learning on tasks that have verifiable answers — math problems, code execution, formal proofs — where a binary reward signal (right/wrong) can be computed automatically. Through this training, models learn to emit an internal monologue before the final answer: a chain of thought that includes tentative steps, self-corrections, dead ends, and restarts. OpenAI calls this reasoning tokens; Anthropic exposes it as thinking blocks in the response object. Either way, you're paying for the deliberation.

The mechanism behind the training is called RLVR (RL with Verifiable Rewards) combined with GRPO. The model generates multiple completions per prompt, a rule-based verifier scores each 0 or 1, and policy weights update based on which completions won. No human labelers, no reward model that can be gamed. DeepSeek-R1 showed in January 2025 that this recipe could match o1's math performance using open-source weights, and the approach has become the dominant post-training stack for reasoning-capable models.

What makes this interesting for production is that the thinking budget is controllable. You can tell Claude to use at most 8,000 thinking tokens on a follow-up clarification and 64,000 on a complex debugging task. OpenAI's o-series maps reasoning_effort to three tiers. This is the knob you'll be turning constantly in production.

flowchart LR
    A[User prompt] --> B[Thinking phase\ninternal CoT tokens]
    B --> C[Final answer\nvisible response tokens]
    B -.billed separately.-> D[(Token meter)]
    C -.billed normally.-> D
    style B fill:#0e7490,stroke:#0e7490,color:#fff
    style D fill:#dc2626,stroke:#dc2626,color:#fff

The thinking phase is billed but typically hidden from the user. Claude's API returns thinking blocks you can log for debugging; OpenAI's o-series summarizes the reasoning trace rather than returning it raw. Neither is a fully faithful explanation of what happened inside the model — the visible trace is real computation, but it's also shaped by training in ways that can make it look more coherent than it is.

Where reasoning models actually win

The performance gains are real, but they're concentrated in a narrow band of tasks.

Multi-step math and formal verification. o3 scored 96.7% on AIME 2024 (as of early 2025). That's not a rounding error over GPT-4 — it's a qualitative capability jump on problems where chained arithmetic errors accumulate and a single wrong turn invalidates everything downstream.

Hard debugging with constraint propagation. When a bug requires reading 400 lines of code, forming a hypothesis, tracing execution across three functions, and rejecting the hypothesis when one counter-example falsifies it — that's the loop reasoning models are trained to do. Standard models pattern-match to common bug signatures; reasoning models can hold more intermediate state and backtrack.

Agentic planning is the third case, whenever plan order matters. If you're building an agent that needs to decide which tools to call, in what order, given partial information, a reasoning model as the planner dramatically reduces the rate of invalid tool sequences. See Your First Production Agent for the agent architecture this slots into.

Code that must satisfy multiple simultaneous constraints. Generate a SQL query that runs in under 200ms, uses only these columns, handles NULLs correctly, and never returns more than 1000 rows. Standard models frequently satisfy three of four. Reasoning models are better at holding all constraints in working memory through the whole generation.

The empirical gap on these tasks is large enough that it's not a close call. But the gap inverts on tasks that don't need deliberation.

Where reasoning models waste money

Classification, extraction, summarization, FAQ answering, and most conversational turns do not benefit from a long internal deliberation. Standard models return correct answers in 2–5 seconds at a token cost that's orders of magnitude lower — roughly 1/300th on the classification math worked out below. Sending these tasks to o3 gives you the same answer after 45 seconds and 8,000 thinking tokens.

The failure is worse than just cost. Reasoning models can actually perform worse on simple, well-defined tasks:

• Format following. A standard model reliably produces JSON matching a schema; a reasoning model may overthink the schema and introduce unexpected keys or restructure arrays.
• Extraction with a clear template. Pull the invoice number, vendor name, and total from this PDF text. The right answer is obvious from the first read. A reasoning model may spend 15,000 tokens exploring edge cases that don't exist in the document.
• Tone-matched responses. Short, warm, casual replies to user messages. Reasoning models have a tendency toward exhaustive coverage in their outputs, fighting the tone requirements.

The underlying issue is that reasoning models are optimized on tasks where more thinking helps. They have no mechanism to detect "this question is trivial" and short-circuit — unless you constrain the budget explicitly or route the request away from them entirely.

{ "type": "model-router", "title": "Standard vs reasoning model routing by task type" }

The cost and latency reality

Let me put numbers to this so it's not abstract.

A support ticket classification request — "is this a billing issue, a technical issue, or a general inquiry?" — might be 200 input tokens and produce 10 output tokens. At mid-2026 GPT-4.1 pricing (~$2/M input, $8/M output, illustrative), that's $0.000480 per call. At 100,000 tickets/day: $48/day.

Route those same tickets to o3 with default reasoning effort. The model generates, say, 8,000 thinking tokens per ticket thinking through edge cases, ambiguous phrasings, and policies it was never given. At o3 reasoning token pricing (~$15/M, illustrative) plus the final output: roughly $0.12 per ticket. At 100,000 tickets/day: $12,000/day. For classification accuracy that's not measurably better.

The intro's tripled bill is this math diluted across the whole traffic mix: the classification slice alone was paying ~250–300x, blended down to 3x by the traffic that actually belonged on the reasoning model.

Back-of-envelope: reasoning model cost on easy tasks

Classification task
  Standard model: 200 input + 10 output tokens
    = $0.00048 per call
    = $48/day at 100k calls

  Reasoning model: 200 input + 8,000 thinking + 10 output tokens
    = $0.12 per call (illustrative mid-2026 o3 pricing)
    = $12,000/day at 100k calls
    = 300x cost multiplier

Hard debugging task
  Standard model: 2,000 input, 500 output
    = $0.008 per call; often fails or needs 3 retries → $0.024 effective cost-per-success
    = For 1,000 sessions/day: $24/day in API costs, but significant developer time
      resolving the sessions that still fail after 3 retries

  Reasoning model: 2,000 input, 40,000 thinking, 800 output
    = $0.62 per call; usually succeeds first try
    = For 1,000 sessions/day: $620/day in API costs
    = 26x higher API cost — but eliminates the developer time spent on failed sessions

The cost picture flips depending on task type. For easy, high-volume tasks the reasoning model is 300× more expensive with no quality gain. For hard, low-volume tasks the reasoning model costs 26× more per API call but can eliminate the developer time and retry overhead that failed standard-model runs require. This is why the decision has to happen per task, not per deployment.

{ "type": "token-cost", "title": "Where the monthly bill comes from: input vs output vs cached tokens" }

Latency tells the same story. GPT-4.1 returns in 2–5 seconds for most prompts. o3 on a hard problem: 30–120 seconds. That's not a model slowdown you tune your way out of. It's an architectural property of generating 40,000 tokens before answering. User-facing features with reasoning models need streaming and thoughtful UX to avoid feeling broken.

How prompting changes

This is the part that surprises most engineers who have spent time tuning prompts for standard models.

Stop writing chain-of-thought instructions. "Think step by step", "reason carefully before answering", "let's work through this methodically" — these are harmful on reasoning models, not helpful. The model is already generating an internal chain. Your instruction either does nothing or anchors the model to a specific reasoning path, preventing it from exploring alternatives it might find more productive. DeepSeek-R1 is particularly sensitive to this: leading the reasoning trace with prescribed steps can cause it to commit early and stop exploring.

Go shorter, not longer. Baroque multi-part prompts optimized for GPT-4 often hurt reasoning models. They can anchor on irrelevant details from a long system prompt. Write concise prompts: role, task, constraints, output format. Let the model fill in the deliberation.

State constraints explicitly. Standard models need soft guidance ("be concise, be accurate"). Reasoning models respond better to hard constraints ("response must be under 300 words", "the answer must be a valid Python 3.11 expression", "if the answer is uncertain, say UNKNOWN rather than guessing"). Hard constraints are checkable, and reasoning models check them.

Zero-shot first, always. Few-shot examples for reasoning models frequently hurt performance. The examples anchor the model to a solution strategy. If your few-shot examples use strategy A and the correct approach for a new problem is strategy B, the model will often use strategy A anyway. Test zero-shot first; only add examples if zero-shot fails on your eval set.

Here's what a prompt redesign looks like in practice:

# Bad: GPT-4 style, verbose CoT prompt
SYSTEM_PROMPT_BAD = """
You are an expert software engineer. When debugging code, you should:
1. First, read through all the code carefully
2. Think step by step about what each part does
3. Consider all possible failure modes
4. Reason carefully about the root cause
5. Only then provide your diagnosis

Think before you answer. Be thorough and methodical.
"""

# Good: Reasoning model style, constraint-focused
SYSTEM_PROMPT_GOOD = """
You are a software engineer debugging Python code.
Task: Identify the root cause of the exception.
Constraints:
- Output format: {"root_cause": "...", "fix": "...", "confidence": "high|medium|low"}
- If you cannot determine the root cause from the provided context, set confidence to "low" and explain what additional information you need.
- Do not suggest fixes you are not confident in.
"""

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-5",  # or claude-opus-4 for harder tasks
    max_tokens=16000,
    thinking={
        "type": "enabled",
        "budget_tokens": 10000  # tune per task; don't leave at max
    },
    messages=[{
        "role": "user",
        "content": "Debug this: " + code_snippet
    }],
    system=SYSTEM_PROMPT_GOOD
)

# thinking blocks and text blocks come back separately
for block in response.content:
    if block.type == "thinking":
        internal_chain = block.thinking  # for logging/debugging only
    elif block.type == "text":
        answer = block.text

For OpenAI's o-series:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="o3",
    reasoning_effort="medium",  # low | medium | high
    messages=[
        {"role": "user", "content": "Debug this: " + code_snippet}
    ]
)

answer = response.choices[0].message.content
# reasoning tokens tracked separately in usage object
thinking_tokens = response.usage.completion_tokens_details.reasoning_tokens

Notice reasoning_effort="medium". That is the single highest-leverage change you can make in production. On tasks that are medium-difficulty rather than research-grade hard, medium effort often recovers 95% of high accuracy at 40% of the cost. Test this on your actual eval set before accepting the default.

The overthinking and underthinking failure modes

Reasoning models fail in two directions, and both are expensive in different ways.

Overthinking happens on ambiguous or trivially easy queries. The model detects ambiguity (or perceives depth that isn't there) and spirals through increasingly elaborate reasoning chains. There's an inverse-U shaped relationship between chain length and accuracy that's been documented across multiple task types: accuracy climbs as reasoning extends, peaks, then falls as the chain becomes so long the model starts second-guessing correct intermediate conclusions. On a missing-premise question — "What's the best database for my use case?" with no use case specified — a reasoning model will spend 20,000 tokens exploring every possible use case rather than asking for clarification. A standard model just asks.

Underthinking is the opposite: on genuinely hard problems, the model switches between reasoning strategies rapidly without committing to any of them. It identifies three plausible approaches, starts down each one for a few hundred tokens, then backtracks. The final answer arrives from an incomplete exploration. This often happens when the thinking budget is too tight for the problem complexity — the model doesn't have token budget to fully resolve a strategy before the budget forces truncation.

{ "type": "error-compound", "steps": 8, "accuracy": 0.9, "title": "Multi-step reasoning: error accumulation across dependent steps" }

The error compounding visualizer above shows why multi-step tasks are where reasoning models earn their keep: at 90% per-step accuracy across 8 dependent steps, end-to-end success is only 43%. Reasoning models improve per-step accuracy by catching and correcting mistakes mid-chain — which is the whole mechanism.

What to do about overthinking in production:

• Route easy queries away. A classifier that sends ticket type, simple factual lookups, and conversational turns to a standard model, and escalates hard debugging/planning tasks to the reasoning model, is the right architecture. The model routing deep-dive covers the mechanics.
• Cap the budget on simple turns. If the user sends "thanks, that worked!", your reasoning model should not spend 5,000 tokens responding. Set budget_tokens=1024 or reasoning_effort="low" for turns you can classify as low-complexity.
• Validate that your question is well-posed. Reasoning models spiral on ill-posed inputs. Before sending to a reasoning model, check that all necessary context is present. If a task has known missing information, the right move is to gather it first, not to let the model hallucinate it.

A decision tree for the routing call

The routing decision is not complicated, but you have to make it consciously rather than defaulting to "just use the best model."

flowchart TD
    A[Incoming task] --> B{Multiple dependent\nlogical steps?}
    B -->|No| C[Standard model\nGPT-4.1 / Sonnet]
    B -->|Yes| D{Is latency critical?\n< 3s response time}
    D -->|Yes| E{Can the result be\nprecomputed or async?}
    E -->|No| C
    E -->|Yes| F[Reasoning model\noff the hot path]
    D -->|No| G{High volume?\n> 1k calls/day}
    G -->|Yes| H{Does eval show\n> 5% accuracy gain?}
    H -->|No| C
    H -->|Yes| I[Reasoning model\nwith budget cap]
    G -->|No| J[Reasoning model\nwith budget cap]
    style C fill:#0e7490,color:#fff
    style I fill:#0e7490,color:#fff
    style J fill:#0e7490,color:#fff
    style F fill:#0e7490,color:#fff

The "does eval show > 5% accuracy gain" branch is the one teams skip. They benchmark the reasoning model on hard examples from the task distribution, see a 20% improvement, and conclude it's worth it for all volume. The right experiment is: run the reasoning model on your actual production query distribution, compute accuracy and cost-per-correct-answer, and compare to the standard model on the same distribution. You will often find the improvement concentrates in 10–20% of queries while the other 80–90% are unchanged or degraded.

The production agent pattern

The model that is best for thinking is rarely the model that should execute individual tool calls. This is the architecture that works:

sequenceDiagram
    participant U as User
    participant R as Reasoning model\n(planner)
    participant E as Standard model\n(executor)
    participant T as Tools

    U->>R: Complex task
    R->>R: Extended thinking\n(plan steps)
    R->>E: Step 1: call tool X with params Y
    E->>T: Tool call
    T-->>E: Result
    E-->>R: Structured result
    R->>R: Update plan\n(brief thinking)
    R->>E: Step 2: call tool Z
    E->>T: Tool call
    T-->>E: Result
    E-->>R: Structured result
    R-->>U: Final answer

The reasoning model generates the plan and makes high-stakes decisions: which tool, in what order, how to handle unexpected results. The standard model executes individual tool calls — it's faster, cheaper, and these individual calls are well-scoped enough that they don't need extended deliberation. Results come back to the reasoning model which updates its plan.

Models like o4-mini collapse this split into a single model by calling tools mid-reasoning — which works, but at reasoning-model prices for every step. The two-model split stays the cost-effective choice when execution steps are trivial, and it's what agentic RAG implementations use when retrieval decisions depend on prior results.

For the cost math: if your agent has a planning step (2,000 tokens thinking) and 8 execution steps (each 500 input / 100 output tokens on a fast model), the split looks like:

Reasoning model (planner): 2,000 thinking + 300 output = ~$0.04
Standard model (executor ×8): 8 × (500 input + 100 output) = ~$0.009
Total: ~$0.05 per agent run

vs. Reasoning model all the way:
8 steps × 2,000 thinking each = 16,000 thinking tokens = ~$0.32
= 6.4x more expensive, no accuracy gain on execution steps

The pattern also gives you a natural place to insert verifiers. Between the planner's output and the executor's action, you can validate the plan against constraints before spending tool call budget on it. See the tool-use round trip article for implementation details.

What breaks in practice

Beyond overthinking, four failure modes show up in production deployments that don't appear in benchmark results:

Benchmark scores don't transfer. o3's AIME score of 96.7% says nothing about its accuracy on your specific codebase debugging task or your domain's legal documents. Always benchmark on your own query distribution before committing to a model. The evaluation module covers how to build that eval set.

Then there's the temptation to trust the trace. The visible thinking blocks or reasoning summary feel like a window into model cognition. They are not. The trace is real computation, but it's shaped by training to be legible, and it can contradict the final answer in subtle ways. A model can write a coherent-looking reasoning trace leading to the wrong answer, and a different trace to the right answer — the trace is not the causal explanation of the output. Don't use it for formal auditability; use it for debugging only.

Distilled small models have different failure profiles. DeepSeek-R1's 7B and 14B distilled variants perform well on math benchmarks but are much more brittle than the 671B parent on adversarial inputs and formatting edge cases. Reasoning capability transferred; robustness did not. If you're using a distilled model for cost savings, test it adversarially.

Budget forcing cuts off mid-thought. If you set budget_tokens=2000 and the model is mid-way through a reasoning chain that needs 4,000 tokens to complete, it will truncate, emit whatever it has, and proceed to a final answer that may be based on incomplete deliberation. Set budgets based on empirical measurement of what the task needs, not arbitrary round numbers. Log the actual thinking tokens used per task type and calibrate from there.

Where to go next

What Are Reasoning Models — the mechanism in depth: how RLVR training produces internal chain-of-thought, what GRPO actually does, and why this is architecturally different from prompted CoT on a standard model.

Prompting Reasoning Models — the full prompting guide: format differences across o3 / DeepSeek-R1 / Claude extended thinking, how to structure system prompts, and the specific patterns that cause models to anchor and stop exploring.

Cost, Latency, and the Model-Selection Decision Tree — worked cost math for production deployments, the full decision tree with volume and accuracy thresholds, and the case for adaptive budgets exposed as a user-facing quality knob.

Reasoning Model Failure Modes — the complete catalog of documented failure modes with reproduction cases and mitigation patterns, including how to detect underthinking at runtime.

Model Routing and Cascades — how to build the classifier that routes easy queries to fast models and hard queries to reasoning models without a manual per-request decision, including cost-quality tradeoff measurements from production systems.

Your First Production Agent — if you want to implement the planner-executor pattern covered above, this module has the scaffolding code and the failure modes specific to the agentic loop.