Cost, latency, and the model-selection decision tree for reasoning

The ticket came in on a Tuesday: the new coding assistant was taking 90 seconds to respond to a request to rename a variable. The model was o3, reasoning effort set to high, applied uniformly to every request. A rename is a grep-and-replace. It does not need 90 seconds of internal deliberation. The monthly token bill was trending toward $180,000, of which a significant fraction was thinking tokens spent on trivial edits. Switching the routing to send rename, comment, and simple lookup tasks to GPT-4.1 while reserving o3 for architecture questions and algorithm design brought the bill to $19,000 and the p95 latency on common operations from 90 seconds to 3 seconds. Same model, same quality on hard tasks — just routed.

That story is not unusual. The 2024-2026 wave of reasoning models (o1, o3, o4-mini, DeepSeek-R1, Claude with extended thinking, Gemini 2.5 Pro) is genuinely powerful on the tasks they're designed for. But they come with a cost structure that punishes naive deployment, and the failure mode is not subtle — you just see enormous bills and irritated users.

What you're actually paying for

A standard LLM request has a predictable token count. You write a prompt, the model generates a response, you pay for both. A reasoning model inserts a third phase: thinking tokens, generated before the visible response, billed as output tokens.

The thinking token count is not fixed. It depends on how difficult the model judges the problem to be. On a hard AIME math problem, o3 might generate 20,000-50,000 thinking tokens before writing its answer. On a simple factual question, it might generate 500. The model decides. You cannot observe this until the request completes, which is one reason billing surprises happen.

As of mid-2026, pricing structures vary by provider, but the pattern is consistent: thinking tokens cost more per token than input but similar to or the same as visible output tokens. Since thinking volume can dwarf the visible response, thinking dominates the bill.

Illustrative mid-2026 pricing (verify with provider; rates drift):

GPT-4.1 (standard):
  Input:  $2 / 1M tokens
  Output: $8 / 1M tokens

o3 (reasoning):
  Input:          $10 / 1M tokens
  Thinking:       $40 / 1M tokens  (billed as output)
  Visible output: $40 / 1M tokens

Claude Sonnet (extended thinking OFF):
  Input:  ~$3 / 1M tokens
  Output: ~$15 / 1M tokens

Claude Sonnet (extended thinking ON):
  Thinking tokens billed as output (~$15/1M)
  Budget controllable: budget_tokens param

DeepSeek-R1 (self-hosted or API):
  API price significantly lower than o3;
  self-hosted on-prem: amortized GPU cost, no per-token fee

The ratio is not the 2x or 3x you might expect from a more capable model tier — it is commonly 10-100x on a per-task basis once thinking tokens are included. The interactive below lets you plug in your own request volume and token mix to see where the monthly bill actually comes from — move the output-token slider and watch it dominate everything else.

{ "type": "token-cost", "title": "Where the monthly bill comes from — output tokens dominate" }

One lever cuts whatever number you land on roughly in half before any routing work: batch APIs. Both major providers discount asynchronous, non-real-time requests by ~50% (illustrative, as of mid-2026), and reasoning-heavy workloads — nightly eval runs, offline code review, bulk document analysis — are exactly the traffic that tolerates hours of turnaround. The ~$0.36 o3 task from the estimate above becomes ~$0.18 through the batch endpoint with zero quality loss, and it pairs naturally with the async job queue you already need for long reasoning latencies.

The latency picture

Thinking tokens are generated autoregressively, just like output tokens, which means the model cannot start returning the visible answer until it has finished thinking. On a hard problem, that produces the kind of response times users associate with server errors, not with normal AI latency.

Observed p50 latencies (as of mid-2026, illustrative — varies by load and problem difficulty):

Model	Simple task	Medium task	Hard task
GPT-4o / GPT-4.1	1-2s	3-5s	5-12s
Claude Sonnet (no thinking)	1-3s	3-6s	6-15s
o3 (reasoning_effort=low)	3-8s	8-20s	15-40s
o3 (reasoning_effort=high)	8-25s	20-60s	60-120s
DeepSeek-R1 (self-hosted)	5-15s	15-40s	40-90s
Gemini 2.5 Pro (thinking on)	4-12s	10-30s	30-80s

The variance matters as much as the mean. At reasoning_effort=high, o3 can take 2 minutes on a genuinely hard problem. Streaming does not help with perceived latency when the first visible token is blocked behind all thinking tokens (unless the provider streams thinking tokens separately, which some now do).

For user-facing applications, anything above 10-15 seconds requires either a loading state with progress indication or asynchronous job queuing. Neither is impossible to build, but both are engineering work that you avoid if you route correctly.

Where reasoning models actually earn their cost

Before deciding the model is too expensive, understand what you're buying. On tasks that require sustained multi-step reasoning, the quality gap is large and real.

flowchart LR
    subgraph "Standard LLM"
        P1[Prompt] --> G1[Direct answer\nno explicit chain of thought] --> R1[Answer\n— may skip steps]
    end
    subgraph "Reasoning Model"
        P2[Prompt] --> T2["Long internal CoT\nwith verification + backtracking\n(thinking tokens)"] --> R2[Answer\n— more reliable on hard tasks]
    end
    style T2 fill:#a855f7,color:#fff

Tasks where the premium is justified:

• Hard algorithmic code: writing a working dynamic programming solution, designing a concurrent data structure, debugging a race condition from a stack trace.
• Mathematical derivation: multi-step proofs, optimization problems, anything where errors in step 3 invalidate steps 4-10.
• Complex planning: agentic tasks requiring the model to sequence tool calls, handle conditional branches, and recover from partial failures.
• Legal and compliance analysis: jurisdiction-spanning questions, contract review requiring systematic checklist traversal.
• Adversarial problem solving: tasks where the model needs to consider and discard wrong approaches before arriving at the right one.

Tasks where the premium is not justified (the common ones):

• Classification, tagging, routing
• Extraction from structured documents
• Summarization of well-defined content
• Translation
• Simple code edits (rename, refactor style, add a docstring)
• Factual lookups where the answer is in the context
• Format conversion

The practical test: could a competent human answer this in under ten minutes without deep deliberation? If yes, the reasoning model is likely generating thinking tokens that produce no better answer than a standard model — you're paying for the process, not the output.

Overthinking: the inverse-U problem

The research on this is clear enough to design around. There is an inverse-U relationship between thinking token count and accuracy on most task distributions: accuracy rises as the model thinks more, peaks, then declines as chains become excessively long. The decline on the right side is not dramatic but it is consistent. Overthinking tends to manifest as:

• The model introduces spurious doubts and reverses a correct early answer
• The model anchors on a framework that doesn't fit and builds on it
• The model generates elaborately justified wrong conclusions

The overthinking pathology is most common on ambiguous or ill-posed inputs. A question with a missing premise can cause a reasoning spiral — the model does not ask for clarification, it reasons under uncertainty in circles. This is one place where standard models handle the input more gracefully, producing a short "I need more information" response rather than a 20,000-token deliberation that concludes nothing.

Mitigation requires both routing and budget control:

import anthropic

client = anthropic.Anthropic()

def call_with_budget(prompt: str, thinking_budget: int = 4000) -> str:
    """
    Cap thinking tokens. Extended thinking is off until you pass a
    budget — so pick it deliberately: a generous cap lets a
    hard-looking prompt burn 20k+ thinking tokens when 4k is enough.
    """
    response = client.messages.create(
        model="claude-sonnet-4-5",
        # On Anthropic, thinking tokens count toward max_tokens,
        # so max_tokens must exceed budget_tokens.
        max_tokens=thinking_budget + 2048,
        thinking={
            "type": "enabled",
            "budget_tokens": thinking_budget,  # thinking cap, must be < max_tokens
        },
        messages=[{"role": "user", "content": prompt}]
    )
    # Extract just the text blocks, not thinking blocks
    return "\n".join(
        block.text
        for block in response.content
        if block.type == "text"
    )

# Low-budget for drafts and iterative work
draft = call_with_budget(prompt, thinking_budget=1024)

# High-budget for final, high-stakes answers
final = call_with_budget(prompt, thinking_budget=16000)

For OpenAI's o-series, the equivalent is the reasoning_effort parameter:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="o3",
    reasoning_effort="medium",   # "low" | "medium" | "high"
    messages=[
        {"role": "user", "content": prompt}
    ],
    max_completion_tokens=4096   # caps both thinking + output combined
)

reasoning_effort=low on o3 is useful for drafts, iterative reasoning, and tasks where you want some deliberation but not the full budget. It reduces latency to the 5-20 second range on medium problems.

Building the routing layer

The most impactful engineering decision is adding a routing classifier between your application and the model. A few-shot classifier that decides "does this task need extended thinking?" costs roughly 200 tokens and 100ms on a small standard model. The savings on misrouted tasks pay for it in minutes.

sequenceDiagram
    participant App as Application
    participant Router as Routing Classifier
    participant Fast as Standard Model
    participant Reason as Reasoning Model

    App->>Router: task + metadata
    Router->>Router: classify complexity\n(fast model, ~200 tokens, 100ms)
    alt simple task
        Router->>Fast: route to GPT-4.1 / Sonnet
        Fast-->>App: response (1-5s, $0.005-0.02)
    else complex task
        Router->>Reason: route to o3 / extended thinking
        Reason-->>App: response (15-60s, $0.10-0.50)
    end

The classifier prompt does not need to be elaborate. Two signals work well in practice:

1. Task type: "Is this classification, extraction, or a simple lookup?" → standard model. "Does this require planning, mathematical reasoning, or generating a correct algorithm?" → reasoning model.
2. User intent signal: If your product exposes a "deep analysis" or "expert mode" toggle, use that directly as the routing signal rather than inferring it.

To build intuition for how much the traffic mix matters, the router simulation below lets you vary the share of hard tasks and the classifier threshold and watch cost and quality move against an all-frontier baseline.

{ "type": "model-router", "title": "Reasoning vs standard model routing — cost and quality tradeoffs" }

An alternative to a classifier is cascade routing: always start with a standard model, evaluate confidence or correctness, escalate to the reasoning model only on failure. This works when you have a fast verifier — either a rule-based check or a ground-truth test case. It adds round-trip latency but eliminates false positives in the routing direction (sending easy tasks to the expensive model).

from openai import OpenAI

client = OpenAI()

def route_with_cascade(
    prompt: str,
    verifier,              # callable(str) -> bool
    fast_model="gpt-4.1",
    slow_model="o3",
) -> str:
    # Try the fast model first
    fast_resp = client.chat.completions.create(
        model=fast_model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=1024,
    )
    answer = fast_resp.choices[0].message.content

    if verifier(answer):
        return answer  # fast path succeeded

    # Escalate
    slow_resp = client.chat.completions.create(
        model=slow_model,
        reasoning_effort="medium",
        messages=[{"role": "user", "content": prompt}],
        max_completion_tokens=4096,
    )
    return slow_resp.choices[0].message.content

For more on building reliable cascades for cost optimization, the model routing and cascades article goes deep on classifier design, fallback chains, and how to avoid the latency penalty of cascades when you don't need them.

The planner-executor split

For agentic workloads, the pattern that survives contact with production is: reasoning model as planner, fast model for execution steps.

flowchart TD
    TASK[Agentic task] --> PLAN[Reasoning model\nGenerates a plan:\nstep 1, step 2, step 3...]
    PLAN --> E1[Fast model\nExecutes step 1\ntool call + result]
    E1 --> E2[Fast model\nExecutes step 2\ntool call + result]
    E2 --> E3[Fast model\nExecutes step 3\ntool call + result]
    E3 --> VERIFY{Verifier\nall steps succeeded?}
    VERIFY -->|Yes| DONE[Final answer]
    VERIFY -->|No| REPLAN[Reasoning model\nReplans from failure point]
    REPLAN --> E1
    style PLAN fill:#a855f7,color:#fff
    style E1 fill:#15803d,color:#fff
    style E2 fill:#15803d,color:#fff
    style E3 fill:#15803d,color:#fff
    style VERIFY fill:#00e5ff,color:#0a0a0f

The reasoning model pays its cost on the planning step, where the quality investment is high-leverage. The execution steps — making a tool call, formatting a result, writing a short API request — do not benefit from extended thinking and should not pay for it. A fast model running those steps at $0.005/call instead of $0.30/call can save 95% of the per-step cost while preserving plan quality.

This architecture maps cleanly onto the agentic workflow patterns covered in the adjacent article in this section.

Comparing models: what actually differentiates them

Benchmark headlines are close enough to be noise for selecting between providers. As of mid-2026, o3, Claude (extended thinking), and Gemini 2.5 Pro all score in the 80-97% range on AIME and GPQA. The numbers have converged. What actually differentiates them in production:

Dimension	What to measure	Notes
Cost per task	Measure thinking tokens on your task distribution	Pricing tiers differ; actual usage varies by problem type
Latency at your percentile	p50 and p99 on your real workload	p99 matters more than p50 for user-facing applications
Context window for thinking	Max thinking tokens allowed	Claude allows up to 128k thinking tokens; o3 varies by tier
Budget control granularity	How fine-grained is the effort/budget API?	budget_tokens (Claude) is continuous; reasoning_effort (OpenAI) is a 3-value enum
Streaming behavior	Does the provider stream thinking tokens or block until done?	Affects perceived latency in chat UIs
Tool use during thinking	Can the model call tools mid-thought?	o4-mini supports this; changes the agentic architecture
Accuracy on your evals	Your task distribution, your evaluator	The only number that matters for production

DeepSeek-R1, as an open-weight model, occupies a different position: lower API cost (when third-party hosted) or no per-token cost (when self-hosted), but at the expense of inference infrastructure management and without the same degree of thinking budget API control. For cost-sensitive, high-volume reasoning workloads where you can staff the infrastructure, it is a realistic option. See the open vs. closed model decision framework for the broader infrastructure tradeoff.

What breaks

The latency contract breaks silently

A product that works at 5 seconds can break at 45 seconds without any code change — just a harder distribution of inputs arriving one day. Reasoning model latency is input-dependent. You need p99 latency monitoring on your real traffic, not just average latency from a controlled test set. Set timeouts generous enough to handle hard tasks but tight enough to fail fast on runaway cases, and stream the response when the provider supports it so the user sees progress.

Budget limits can hit mid-reasoning

When you cap budget_tokens or use reasoning_effort=low, the model can hit the limit mid-thought and be forced to answer with incomplete reasoning. The answer quality degrades gracefully in most cases — the model usually produces something plausible — but on high-stakes tasks (medical dosing, financial calculations, security reviews) this is a failure mode you need to test explicitly. Validate that your chosen budget is sufficient for the hardest cases in your distribution before deploying to production.

Thinking tokens are not transparent

The internal chain of thought is often hidden from you (some providers expose it, some do not, and even when exposed it is not a faithful trace of what the model computed). The thinking trace can be internally inconsistent with the final answer — the model can reason itself toward conclusion A and then output conclusion B. Do not use the thinking trace as an explanation or audit trail. If you need an auditable reasoning trace, structure the output explicitly in the visible response.

Distilled models transfer patterns, not robustness

Small models fine-tuned on R1 traces (1.5B-14B parameter reasoning-capable models) are genuinely capable on the narrow tasks represented in their training distribution. They fail badly on adversarial inputs and out-of-distribution problems that the parent model handles, because they learned the pattern of reasoning without the deep weight representations that let the parent handle novel adversarial variation. If you use a distilled reasoning model in production, test it specifically on adversarial and edge-case inputs from your real distribution, not just on in-distribution benchmarks.

Majority voting amplifies bias at scale

If you use parallel sampling (generate N solutions, take the majority vote) to improve accuracy without increasing per-solution compute, watch for the failure mode at large N: when the model has a systematic bias toward a wrong answer, majority voting amplifies that bias rather than correcting it. You need a reliable verifier — a rule-based checker, a test suite, or a process reward model — to catch this. Majority voting alone is not sufficient on tasks with systematic failure modes. The test-time compute scaling strategies article covers this in detail.

The judgment call

There is no single right answer on model selection, but there is a right process:

1. Characterize your task distribution first. What fraction of incoming tasks genuinely require multi-step deliberation? For most products, it is 5-20%, not 80%.
2. Run both models on your real tasks with your real evaluator. Not AIME. Not GPQA. Your tasks. Measure accuracy and cost per correct answer.
3. Set explicit budgets in production. Don't accept the provider default — o3 defaults to reasoning_effort=medium and Claude requires an explicit budget_tokens to think at all, and neither default reflects your cost-accuracy tradeoff. Start with reasoning_effort=low or budget_tokens=4000 and only increase if your evals show accuracy dropping.
4. Route asynchronous work through the batch API. Evals, offline analysis, scheduled reports — anything that doesn't need a synchronous answer gets ~50% off (illustrative) for one line of code. It is the cheapest half you will ever save.
5. Build the routing layer before you need it. A 100ms classifier saves 90% of reasoning model cost on a typical production workload. It is easier to build when you're designing the system than to retrofit when the bill arrives.
6. Measure the right thing. The useful metric is cost-per-correct-answer on your task distribution, not accuracy alone and not cost alone. A model that costs 3x more but has 2x the accuracy might be the right tradeoff — but only if accuracy matters more than cost for that task, and only if the accuracy gap holds on your inputs.

If you're designing a new system and uncertain where to start: begin with a standard model (GPT-4.1, Claude Sonnet, Gemini Flash) for the full surface area, measure accuracy, identify the failure modes, and add a reasoning model for the specific failure cases that justify the cost. This is more work than defaulting to o3 for everything, but it produces a system you can actually explain on a billing review.

For deeper coverage of failure modes specific to reasoning models — overthinking, underthinking, missing-premise spirals, sycophantic reasoning chains — the limits and failure modes article is the companion read. For prompting differences that affect cost indirectly by making reasoning more efficient, prompting reasoning models has the mechanics.