What Are Reasoning Models? Chain-of-Thought, o1, and the Inference-Time Shift

In September 2024, GPT-4o could solve around 12% of problems on AIME — the American Invitational Mathematics Examination that trips up most high school students. o1 solved 83% of the same problems. Not by being larger. Not by being trained on more data. By spending more compute at inference time, thinking before answering.

That gap — 12% to 83% on the same hardware generation — is the reasoning model story in one number. The question worth understanding is how that compute gets spent, because it changes how you should prompt these models, structure your pipelines, and pay for them.

What "reasoning" actually means here

The word gets abused. Every language model does something that could loosely be called reasoning — attending to context, making implicit inferences, stringing together related facts. What the o1/DeepSeek-R1 family does is qualitatively different: it generates an extended internal chain-of-thought as a deliberate, learned behavior, allocating additional compute to harder problems before committing to an answer.

The mechanism is not prompting. You cannot get this behavior by adding "think step by step" to a standard LLM. Prompted chain-of-thought is a useful technique — it can meaningfully improve accuracy on GPT-4 — but the model generating it is still just predicting the next token from your instructions. There's no learned connection between "I generated CoT" and "my answer will be better."

Reasoning models learned that connection through reinforcement learning. The training setup, which we cover in depth in How Reasoning Models Are Trained, works roughly like this: the model generates multiple candidate reasoning traces and final answers; a rule-based verifier checks whether the answer is correct (0 or 1); the RL algorithm updates the model weights to make it more likely to generate the kind of traces that led to correct answers. Over millions of such updates, the model discovers — on its own, without human-labeled chains — that self-checking, backtracking, trying alternative approaches, and verifying intermediate results lead to better outcomes.

The result is a model that does something resembling deliberation: generating intermediate steps, noticing when an intermediate result seems off, revising, and arriving at an answer only when it's been internally validated.

sequenceDiagram
    participant U as User prompt
    participant T as Internal thinking
    participant A as Final answer

    U->>T: "Solve: if 3x + 7 = 22, find x^2 + 2x"
    Note over T: tries direct algebra...
    Note over T: x = 5, checks: 3(5)+7=22 ✓
    Note over T: computes x²+2x = 25+10 = 35
    Note over T: double-checks arithmetic...
    T->>A: "35"
    Note over A: Internal trace hidden or summarized

The thinking trace is typically hidden from the final API response, or returned as a separate block. This is not just a product decision — the internal reasoning is not a faithful explanation of what the model computed. Research consistently shows that visible CoT in these models is partially post-hoc: the visible trace and the final answer can be internally inconsistent. Trust the answer, not the trace, as ground truth.

The key models and where they sit

OpenAI o1 (September 2024) was the first widely-deployed reasoning model. It introduced the model class to practitioners and established the basic UX: longer latency, higher cost, dramatically better results on hard problems. The series continued with o3, which introduced adaptive compute budgets — the same model can spend more or less thinking time per request via a reasoning_effort parameter (low/medium/high).

DeepSeek-R1 (January 2025) is the open-source watershed. The purest version of the recipe is its precursor, R1-Zero: GRPO (Group Relative Policy Optimization) applied directly to the base model with binary 0/1 rewards from math and code verifiers — no SFT warm-up, no human-labeled chain-of-thought, no separate reward model. R1-Zero reasoned well but wrote badly (language mixing, unreadable traces), so R1 proper added a cold-start SFT phase on a few thousand curated long-CoT examples before GRPO, with rule-based accuracy rewards plus format and language-consistency rewards. The result matched o1 on AIME and Codeforces benchmarks. More importantly, they released the model weights and the training methodology, which the community immediately used to distill reasoning capability into 1.5B to 14B parameter models. Those distilled models can run on a laptop and exhibit reasoning behavior that was unachievable at that scale six months earlier.

Claude extended thinking (Claude 3.7 Sonnet and later) exposes the thinking budget directly: set budget_tokens in the API and the model will spend up to that many tokens on internal deliberation. The ceiling is 128k thinking tokens per request — which, at roughly 1,000 tokens per second of model time, translates to over two minutes of thinking on a hard problem.

Gemini 2.5 Pro integrated thinking with a 1M token context window, which makes it particularly useful for tasks that require reasoning over large documents. As of mid-2026, it outperforms most competitors on AIME and GPQA benchmarks, though benchmark compression means those numbers are less informative than they used to be.

flowchart TD
    O1["o1 (Sep 2024)\nFirst deployed reasoning model\nHidden thinking, fixed compute"] --> O3["o3 (early 2025)\nadaptive reasoning_effort\n96.7% AIME 2024"]
    O3 --> O4["o4-mini (2025)\nReasoning + native tool use\nParallel function calls in thinking"]
    DSR0["DeepSeek-R1-Zero\nPure RL from base model\nNo SFT warm-up"] --> DSR1["DeepSeek-R1 (Jan 2025)\nOpen weights, GRPO\nMatches o1 on math/code"]
    DSR1 --> DIST["Distilled: R1-7B, R1-14B\nReasoning on consumer hardware"]
    C37["Claude 3.7 Sonnet\nextended thinking\nbudget_tokens API param\nup to 128k thinking tokens"]
    G25["Gemini 2.5 Pro\n1M context + thinking\nTop AIME/GPQA scores mid-2026"]

    style O3 fill:#00e5ff,color:#0a0a0f
    style DSR1 fill:#a855f7,color:#fff
    style C37 fill:#0e7490,color:#fff

What the inference-time compute shift actually means

The phrase "scaling inference compute" sounds abstract. Here's what it means concretely.

Traditional LLM scaling laws say: if you want a better model, train a bigger model on more data. This is expensive upfront (billions of dollars for frontier models) but cheap at inference time — every request costs roughly the same compute regardless of problem difficulty.

Reasoning models invert this. The model can be the same size as a standard one, but it allocates variable compute per request. Easy question? Short internal chain, fast answer, low cost. Hard proof? Long chain, many self-checks, minutes of thinking, high cost. The quality improvement comes from inference-time compute, not training-time compute — at least for the class of problems where deliberation helps.

This creates a new design dimension: thinking budget. You can now tune the cost-quality tradeoff per request, not just per model choice. A customer support bot handling routine refund requests needs zero thinking budget. The same system handling a complex multi-party contract dispute might warrant several thousand thinking tokens. Leaving the budget at its maximum default on every request — including trivial follow-up turns — is one of the most common and expensive mistakes in production deployments.

{ "type": "token-cost", "title": "Reasoning vs standard model cost waterfall" }

The prompting rules change

This is where most teams get tripped up when migrating from standard LLMs to reasoning models. Techniques that help on GPT-4 either do nothing or actively hurt on o3 and Claude extended thinking.

Do not add "think step by step." The model is already doing this internally. Adding the instruction doesn't increase the thinking budget — it wastes tokens in your prompt and can anchor the model to a specific reasoning pattern that isn't optimal for the problem.

Fewer few-shot examples, not more. Reasoning models often perform best zero-shot. Including examples doesn't seed the model's reasoning the same way it does for a standard LLM; instead, the examples can pattern-match the model onto the wrong solution strategy. Start without examples, add them only if you observe a clear quality improvement on your eval set.

State constraints explicitly. Tell the model what the output format should be, what counts as a correct answer, what constraints matter. Don't rely on the model inferring this from context. A reasoning model will reason within whatever frame you give it — if the frame is ambiguous, it will pick one and commit to it.

For DeepSeek-R1 specifically: avoid leading the reasoning trace with specific reasoning steps in your prompt. The model can anchor on them and stop exploring alternatives. This is different from Claude and o3, which are more resistant to this anchoring.

We cover all of these in depth — with real prompt examples and ablation results — in Prompting Reasoning Models: What Changes and What Breaks.

Calling the API

Here's what a reasoning model call looks like with current SDK idioms. The thinking content block is separate from the final text response, and you can choose whether to display it.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=16000,
    thinking={
        "type": "enabled",
        "budget_tokens": 10000,  # up to 128k; tune per task complexity
    },
    messages=[{
        "role": "user",
        "content": "A company has revenue of $2.4M in Q1, growing 15% QoQ. "
                   "Fixed costs are $800k/quarter, variable costs are 40% of revenue. "
                   "At what quarterly revenue does EBITDA turn positive, "
                   "and how many quarters away is that at the current growth rate?"
    }]
)

# The response contains both thinking and text blocks
for block in response.content:
    if block.type == "thinking":
        print(f"[thinking: {len(block.thinking)} chars]")
    elif block.type == "text":
        print(block.text)

With OpenAI's o3:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="o3",
    reasoning_effort="high",  # "low" | "medium" | "high"
    messages=[{
        "role": "user",
        "content": "Write a Python function that solves the N-Queens problem "
                   "using constraint propagation and backtracking. "
                   "Optimize for readability over raw speed."
    }]
)

# usage includes reasoning tokens
print(response.choices[0].message.content)
print(f"Reasoning tokens used: {response.usage.completion_tokens_details.reasoning_tokens}")

The reasoning_tokens field in usage is important. That's the cost you're paying for the thinking, and it's often the majority of your bill on hard problems. Instrument this and track it per request type from day one.

What breaks — and when reasoning models are the wrong choice

Reasoning models fail in predictable ways, and knowing them ahead of time prevents expensive mistakes.

Overthinking easy problems. Give an o3-class model a simple lookup question — "What's the capital of France?" — and it might generate 500 thinking tokens reassuring itself that Paris is indeed the capital. The accuracy is fine; you just paid 50x more than necessary. At high volume, this is a real cost. The mitigation is routing: a classifier that detects query complexity and sends easy questions to a fast, cheap model. We cover the full routing decision tree in Cost, Latency, and the Model-Selection Decision Tree for Reasoning.

Underthinking genuinely hard problems. The inverse pathology: on certain types of hard problems (particularly ones with missing premises or unusual structure), reasoning models switch between strategies without committing. The thinking trace looks busy but doesn't converge. You get a confident-sounding wrong answer. This is harder to detect and arguably more dangerous than overthinking.

Spiraling on ill-posed questions. Ask a reasoning model "what is the best approach?" without specifying best according to what criteria, and it will pick criteria and reason toward an answer that is internally consistent but may be useless for your actual purpose. Standard LLMs handle this more gracefully — they pattern-match to a plausible interpretation and answer. Reasoning models commit more fully to their chosen frame. The fix is specificity: state success criteria explicitly.

Trusting the visible trace. The internal chain-of-thought is partially post-hoc rationalization. Research on o1-class models consistently finds cases where the visible reasoning trace supports the wrong answer but the final answer is correct, and vice versa. Do not use the reasoning trace as an explanation for auditing purposes without treating it skeptically. The trace is useful for debugging and qualitative understanding — it's not ground truth for what the model actually computed.

Benchmark scores don't transfer. o3 hits 96.7% on AIME 2024. This does not predict anything about its performance on your specific task distribution. Teams that deploy reasoning models based on headline benchmark scores without running their own evals regularly discover that the model underperforms on their domain-specific tasks. Always benchmark on representative samples of your actual workload before committing to a model choice.

For a full taxonomy of failure modes — with reproduction examples and mitigation patterns — see Limits and Failure Modes: Overthinking, Underthinking, and When Reasoning Breaks.

The model landscape as of mid-2026

The benchmark scores on AIME and GPQA have largely converged across frontier providers. As of mid-2026, multiple models score above 90% on AIME; the differentiation has shifted to:

Model	Thinking budget control	Context window	Notable strength	Relative cost
o3 (OpenAI)	reasoning_effort enum	200k	Adaptive compute, native tool use in o4-mini	High
o4-mini (OpenAI)	reasoning_effort enum	200k	Reasoning + parallel tool calls, lower cost than o3	Medium
Claude extended thinking	budget_tokens integer	200k	Finest-grained budget control, strong on long docs	High
Gemini 2.5 Pro	thinking_config	1M	Largest context, top AIME scores	High
DeepSeek-R1 (self-hosted)	Prompt-based, some inference servers expose budget	128k	Open weights, distillable, cost-efficient at scale	Variable (hosting cost)
R1 distilled (7B–14B)	N/A	128k	Edge/on-prem deployment, near-R1 on math	Very low

Prices drift fast — treat the "relative cost" column as a rough ordering, not fixed numbers.

How this fits into your architecture

The most common production pattern is not "replace all LLM calls with reasoning models." It's a tiered architecture:

flowchart TD
    REQ[Incoming request] --> CLASS[Complexity classifier]
    CLASS -->|"Simple: lookup,\nformat, classify"| FAST["Fast model\n(GPT-4.1, Claude Haiku)\n~$0.001/request, ~1s"]
    CLASS -->|"Medium: multi-step,\nsome analysis"| MED["Standard frontier\n(GPT-4o, Claude Sonnet)\n~$0.01/request, ~3s"]
    CLASS -->|"Hard: proof,\nplanning, complex code"| REASON["Reasoning model\n(o3, Claude extended thinking)\n~$0.50/request, ~30-120s"]

    style FAST fill:#15803d,color:#fff
    style MED fill:#00e5ff,color:#0a0a0f
    style REASON fill:#a855f7,color:#fff

The classifier itself should be cheap and fast — a small model or a rules-based heuristic based on query length, topic, and user tier. You're routing maybe 5–15% of requests to the expensive tier. The result is near-reasoning-model quality on hard problems at a blended cost much closer to standard model pricing.

A second common pattern is the planner-executor split: a reasoning model handles orchestration and planning (breaking a complex task into steps, deciding what tools to call), while a cheaper standard model handles execution of individual steps. This is covered in detail in Reasoning Models Plus Tools: Agentic Workflows and Function Calling and Test-Time Compute Scaling: Sequential, Parallel, and Tree-Based Strategies.

The judgment call

When do you actually reach for a reasoning model? Two tests work in practice.

The 10-minute rule: if a human expert would spend more than ten minutes thinking through the problem carefully — not because of data lookup time, but because the reasoning itself is complex — a reasoning model is likely to be worth the cost. Math proofs, intricate code generation, complex legal or financial analysis, multi-step agentic planning with many dependencies. Below ten minutes of expert deliberation, a standard model with good prompting is usually sufficient.

The error-compounding test: does getting step N wrong predictably destroy step N+1? If so, you want the model to self-check during generation, which is exactly what reasoning models train to do. A standard LLM generates step N+1 without reconsidering step N; a reasoning model will revisit and correct. For linear tasks where steps are independent, this doesn't matter. For dependent chains, it does.

The flip side: reasoning models are the wrong choice when you need sub-second latency, when you're handling millions of requests per day at marginal cost, or when your task is something a standard model handles well. Spending 10 seconds and $0.50 to extract an entity from a support ticket is not improving your product — it's lighting money on fire.

The 2024–2026 inference-time compute shift is real and consequential. But the correct response is not to route everything to the most powerful reasoning model you can find. It's to build the classification infrastructure that sends each query to the right tier — and to understand your task distribution well enough to know where the line is. That's the actual engineering work.

Further reading

• Prompting Reasoning Models: What Changes and What Breaks — the concrete prompting rules, with examples
• Test-Time Compute Scaling: Sequential, Parallel, and Tree-Based Strategies — the algorithmic levers behind inference-time compute
• Cost, Latency, and the Model-Selection Decision Tree for Reasoning — worked numbers and a routing framework
• How Reasoning Models Are Trained: RLVR, GRPO, and the Post-Training Stack — the RL mechanics behind o1 and R1
• Limits and Failure Modes: Overthinking, Underthinking, and When Reasoning Breaks — failure taxonomy with reproduction examples
• Few-Shot Prompting, Chain-of-Thought, and When Reasoning Models Flip the Script — how prompted CoT relates to and differs from trained reasoning
• The Model Landscape in 2025-2026 — where reasoning models fit in the broader provider landscape
• DeepSeek-R1 technical report (arXiv:2501.12948) — the open-source training methodology
• OpenAI reasoning best practices (developers.openai.com) — official guidance on prompting o-series models