Prompting reasoning models: what changes and what breaks

A team I know spent two weeks porting their legal analysis pipeline from GPT-4o to o3. Their existing prompts were refined: multi-paragraph system prompt, four carefully chosen few-shot examples, explicit "reason step by step" instruction, intermediate reasoning checkpoints. On GPT-4o, this prompt had reached 87% accuracy on their internal eval set. On o3, it dropped to 79%.

They stripped the prompt to three sentences — problem statement, output format, and one critical constraint — and went zero-shot. Accuracy: 91%.

That experience isn't unusual. Understanding why requires understanding what reasoning models actually do, and what your prompt is now competing with.

What the model is already doing

Standard LLMs — GPT-4o, Claude Sonnet without extended thinking, Llama 3.3 — generate tokens one at a time, each conditioned on what came before. If you want the model to reason carefully, you need to elicit that in the prompt: "think step by step" works because it shifts the probability distribution toward tokens that resemble reasoning chains.

Reasoning models like o3, o4-mini, DeepSeek-R1, and Claude with extended thinking were trained differently. They learned via reinforcement learning with verifiable rewards (RLVR) to generate extended internal thought before producing their final answer. The mechanism is described in depth in what are reasoning models, but the practical consequence for prompting is this: the model's default behavior on a hard question is already to explore multiple approaches, backtrack, and verify before committing. You didn't build that — it was trained in.

When you add "think step by step" or "reason carefully," you're not turning on reasoning. You're injecting text at the start of what the model was already going to do. In the best case, your instruction is ignored. In the worse case — which happens more than you'd expect — it anchors the model to a specific approach before it has explored alternatives, cutting off better solutions it would have found on its own.

sequenceDiagram
    participant U as Your prompt
    participant M as Reasoning model
    participant T as Internal thinking trace
    participant A as Final answer

    Note over U,A: What happens with a standard "think step by step" prompt
    U->>M: "Think step by step. Step 1: ..."
    M->>T: Anchors thinking to your prescribed steps
    T->>T: May miss alternative approaches
    T->>A: Answer constrained by your scaffolding

    Note over U,A: What happens with a constraint-focused prompt
    U->>M: "Problem statement + constraints + output format"
    M->>T: Generates its own reasoning strategy
    T->>T: Explores alternatives, backtracks, verifies
    T->>A: Answer from unconstrained search

The zero-shot default

The most reliable prompting shift for reasoning models is to go zero-shot first. Not "add a zero-shot attempt and then add examples if it fails" — actually default to zero-shot and treat examples as a last resort.

This is the opposite of best practice for standard LLMs. Why does it work here?

Few-shot examples teach a model how to solve problems by demonstration. That's valuable when the model doesn't have a solving strategy. Reasoning models have one — it's their training. Your examples now compete with their internal strategy. If your example's approach happens to be suboptimal for the current problem, the model may follow it anyway. A reasoning model that would have found a more elegant proof on its own gets anchored to the approach your three examples used.

The one exception: format examples. If you need a very specific output structure — JSON with particular fields, a markdown table with exact column names, a response under a hard character limit — showing one example of the desired format is fine. You're not demonstrating solution strategy; you're demonstrating output contract.

# Pattern that often HURTS on reasoning models
system_prompt_bad = """
You are an expert code reviewer. Think step by step.
Here are some examples:

Example 1:
Code: [long example]
Review: [detailed chain-of-thought walkthrough]

Example 2:
Code: [another example]
Review: [another chain-of-thought walkthrough]

Now review the following code step by step:
"""

# Pattern that tends to WORK better
system_prompt_good = """
You are a code reviewer. Identify bugs, security issues, and performance problems.

Output format:
{
  "severity": "critical|high|medium|low",
  "issues": [{"line": int, "type": str, "description": str, "fix": str}],
  "summary": str
}

Respond only with valid JSON.
"""

# The model's internal reasoning handles the "how" — your job is the "what"

What to put in the prompt instead

If you're removing CoT scaffolding and examples, what fills the space? Constraints, not process.

State the output format precisely. This matters more for reasoning models than standard ones because the o-series API returns plain text by default — markdown is off. If downstream code expects a JSON object, ask for JSON and give the schema. If the response must be under 200 words, say so.

Hard constraints belong in the prompt verbatim, not left for the model to infer from context. "The function must handle null inputs." "The legal analysis must cite only statutes in force as of 2024." "The budget cannot exceed $500." Reasoning models are good at complex inference, but they spend their thinking budget on what you make salient — a buried constraint is a skipped constraint.

And define what success looks like — not how to get there, but what "correct" means for your specific problem. This is where the difference from a CoT prompt is sharpest: you're not prescribing the reasoning process, you're defining the target.

import anthropic

client = anthropic.Anthropic()

# Extended thinking with explicit budget and format constraints
response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=8000,
    thinking={
        "type": "enabled",
        "budget_tokens": 5000  # Picked from an eval — not copied from a docs example
    },
    messages=[{
        "role": "user",
        "content": """Review this contract clause for ambiguity that could create liability.

Clause: "The vendor shall deliver services within a reasonable timeframe as mutually agreed."

Constraints:
- Identify every undefined term that would require litigation to resolve
- Assess severity: critical (likely to be disputed) / moderate / low
- Propose replacement language for each critical issue

Output as JSON:
{
  "undefined_terms": [{"term": str, "severity": str, "risk": str}],
  "replacements": [{"original": str, "proposed": str, "rationale": str}],
  "overall_risk": "high|medium|low"
}"""
    }]
)

Note the budget_tokens: 5000 in the extended thinking config. Claude makes you pick this number — there is no default to fall back on — so pick it from an eval on your own tasks, not from whatever the docs example used.

The thinking budget is a cost dial, not a quality guarantee

Claude's extended thinking API accepts budget_tokens from 1024 to 128000. OpenAI's o3 accepts reasoning_effort as "low", "medium", or "high". These are not binary on/off switches — they control how much compute the model spends searching before it commits to an answer.

The tempting mistake is to always set the budget to maximum. The logic seems sound: more thinking should give better answers. The reality is more nuanced, as described in reasoning models cost and latency tradeoffs. Research on chain-of-thought length (Wu et al., 2025) shows an inverse U-shaped relationship between reasoning length and accuracy — chains that are too short miss important steps, but chains that are too long tend to spiral and actually degrade on some task types. The optimal budget varies by task difficulty, not by "more is always better."

Thinking budget calibration by task type (illustrative, tune on your own data):
────────────────────────────────────────────────────────────────────────────────
Simple extraction, classification, summarization:
  → budget_tokens: 1024–2000
  → Matches full-budget accuracy on most tasks; 10-30× cheaper

Moderate reasoning (debugging, refactoring, short proofs):
  → budget_tokens: 4000–8000
  → Sweet spot for most engineering tasks

Hard math, complex legal analysis, multi-step planning:
  → budget_tokens: 16000–32000
  → Accuracy gains start plateauing above this for most problems

Research-level math, adversarial security analysis:
  → budget_tokens: 64000–128000
  → Justified only when the accuracy delta at this level matters

Latency scales with the budget: roughly 2–5 s of extra wall-clock time at a
2000-token budget vs. 30–90 s at 32000 (illustrative). Thinking tokens are
generated serially, so the budget is a latency dial as much as a cost dial.

A practical pattern for production: run a small eval on your specific task distribution at budget levels 2000, 8000, 32000, and 128000. The accuracy-cost curve usually has a clear knee. Ship at that knee, not at the maximum.

from openai import OpenAI

client = OpenAI()

# o3 with controlled reasoning effort — use "low" for well-defined, fast tasks
response = client.chat.completions.create(
    model="o3",
    reasoning_effort="low",  # "low" | "medium" | "high"
    messages=[
        {
            "role": "developer",
            "content": "You review Python code for security vulnerabilities. Output JSON only."
        },
        {
            "role": "user", 
            "content": f"Review this function:\n\n```python\n{code_snippet}\n```\n\nJSON schema: {{\"vulnerabilities\": [{{\"line\": int, \"type\": str, \"severity\": str}}]}}"
        }
    ]
)

Note the developer role instead of system — this is the o-series convention for instructions that should persist across a conversation. The API accepts both, but developer is the canonical form for o3 and later.

DeepSeek-R1 specifics

DeepSeek-R1 is the open-source reasoning model most teams reach for when they want to self-host or avoid OpenAI/Anthropic pricing. The prompting dynamics are similar to o3 but with a few idiosyncrasies worth knowing.

R1 exposes its thinking trace directly in the output, wrapped in <think>...</think> tags before the final answer. This is different from Claude's extended thinking (returned in a separate thinking block, and on Claude 4.x models summarized rather than raw — you're still billed for the full internal trace) and from o3 (where the reasoning trace is not returned to the caller by default). The R1 trace is the actual generation — you can read every token of it, which is genuinely useful for debugging.

The main R1-specific pitfall: don't seed the thinking trace. Some prompts try to start the model's reasoning with a specific line — "First, I should consider..." — by including that text before the <think> block closes. This anchors the model to your prescribed starting point and prevents it from exploring the space naturally. The model's self-generated first step is almost always better than yours for hard problems.

# For DeepSeek-R1 via API — let the thinking emerge naturally
import os
from openai import OpenAI

# DeepSeek's API is OpenAI-compatible, but you must point the client at it
client = OpenAI(
    base_url="https://api.deepseek.com",
    api_key=os.environ["DEEPSEEK_API_KEY"],
)

response = client.chat.completions.create(
    model="deepseek-reasoner",  # R1 model ID
    messages=[
        {
            "role": "system",
            "content": "You solve coding problems. Return only the function implementation, no explanation."
        },
        {
            "role": "user",
            "content": "Implement a function that finds the longest palindromic substring in O(n) time.\n\nRequirements:\n- Pure Python, no external libraries\n- Handle empty strings and single characters\n- Return the substring, not its indices"
        }
        # Do NOT add: {"role": "assistant", "content": "<think>\nFirst, I'll consider..."} 
        # This pre-fills the thinking trace and anchors the model
    ]
)

# The response will contain reasoning_content (the <think> block) separately
reasoning = response.choices[0].message.reasoning_content
answer = response.choices[0].message.content

What breaks: the failure modes specific to reasoning model prompting

Spiral on ill-posed inputs

Standard models on an ambiguous or underspecified question will typically produce something — often a hedge, often wrong, but usually fast. Reasoning models on the same question may spiral. The extended thinking budget gets spent exploring interpretations, the model generates multiple conflicting analyses, and the final answer is long, uncertain, and expensive to produce.

The mitigation is upfront constraint specification. If a question has missing premises — "which option is better?" without defining "better" — state explicitly what the model should assume or ask for clarification before proceeding.

Format regressions from o-series

OpenAI's o-series models do not return markdown by default in API responses (as of o1-2024-12-17 and later). If your downstream code renders markdown, parses headers, or expects bullet-point lists, you need to explicitly request formatted output. A system prompt addition like "Format responses in Markdown with headers and bullet points" is sufficient, but it has to be there.

This catches teams porting from GPT-4o where markdown rendering was the default, and they only discover the regression when frontend rendering breaks in production.

Porting gotchas: sampling and multi-turn thinking

Two more surprises hit teams moving an existing pipeline onto Claude extended thinking. First, enabling thinking constrains sampling: the API requires temperature at 1 and rejects top_k (and most top_p values) alongside a thinking block. If your GPT-4o pipeline carefully tuned temperature=0.2 for deterministic extraction, that setting doesn't carry over — the request errors until you drop it. Budget a re-tuning pass instead of assuming your sampling config transfers.

Second, multi-turn and tool-use conversations must pass the assistant's thinking blocks back unmodified in subsequent requests — Anthropic's API validates this during tool use. Pipelines that persist only the text portion of each turn and reconstruct messages from it will fail on the next request. Store the full content blocks, thinking included.

The failure mode: you have three examples that all solve a certain category of problem using approach A. The current problem would be better solved with approach B. The reasoning model's own search starts down path B — but your in-context examples keep pulling generation back toward A, and it switches.

On standard models, this is usually fine — you're helping by providing examples. On reasoning models, your examples compete with the model's own search, and when they conflict, the result is inconsistency. The model may produce a hybrid that combines both approaches badly.

flowchart TD
    START["New problem arrives"]
    START --> ZS["Zero-shot first"]
    ZS --> EVAL{"Accuracy sufficient\non your eval set?"}
    EVAL -->|Yes| SHIP["Ship zero-shot prompt"]
    EVAL -->|No| FMT{"Is the failure about\noutput format?"}
    FMT -->|Yes| FMTEX["Add one format example\n(not solution strategy)"]
    FMTEX --> EVAL2{"Accuracy sufficient?"}
    EVAL2 -->|Yes| SHIP
    EVAL2 -->|No| BUDGET["Try higher thinking budget"]
    FMT -->|No| CONSTR["Add explicit constraints\nand success criteria"]
    CONSTR --> EVAL3{"Accuracy sufficient?"}
    EVAL3 -->|Yes| SHIP
    EVAL3 -->|No| BUDGET
    BUDGET --> DISTRIB{"Check task distribution —\ncould a standard model\nroute these?"}
    style SHIP fill:#15803d,color:#fff
    style BUDGET fill:#ffaa00,color:#0a0a0f

Sycophantic reasoning traces

Reasoning models can generate a long, plausible-sounding chain that arrives at an answer the user seemed to want — even when that answer is wrong. The trace looks thoughtful; the conclusion is wrong. This is more insidious than a standard model hallucination because the apparent deliberation creates false confidence.

This is covered in depth in reasoning model failure modes. For prompting, the mitigation is adversarial prompting in evals — include test cases where the "obvious" answer is wrong and measure whether the model arrives at the correct but non-obvious answer. Sycophantic reasoning will fail these; honest reasoning will not.

Reading the reasoning trace

When the trace is visible — R1's <think> block, Claude's thinking content block — it tells you things the final answer doesn't. One caveat before relying on it: Claude 4.x thinking blocks are summarized, not the raw internal trace, so treat phrase-level pattern matching on them as best-effort. On R1, where the trace is the raw generation, the same patterns are much more dependable.

A trace that shows confident exploration followed by a clean commitment is a good sign. A trace that shows repeated backtracking, conflicting intermediate conclusions, or statements like "but actually I'm not sure" followed by a confident final answer is a warning sign. The model committed to an answer it didn't fully work out.

This is especially useful for debugging wrong answers. When a reasoning model gets something wrong, the trace usually shows you where the reasoning went off the rails — a wrong assumption at step 3, a calculation error at step 7. That's actionable in a way that a standard model's wrong answer often isn't. You can fix the prompt by surfacing the constraint the model missed, rather than guessing from the output alone.

# Using Claude's thinking block for debugging
import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=10000,
    thinking={"type": "enabled", "budget_tokens": 8000},
    messages=[{"role": "user", "content": your_problem}]
)

for block in response.content:
    if block.type == "thinking":
        thinking_text = block.thinking
        # Red flags to grep for (heuristic — Claude's summarizer may not
        # preserve these verbatim; on R1's raw trace they're reliable):
        # - "but actually"
        # - "I made an error"
        # - "let me reconsider" appearing more than 2-3 times
        # - Contradictory conclusions ("X is true" followed by "X is false")
        suspicious = any(phrase in thinking_text.lower() for phrase in [
            "but actually", "wait, i", "i made an error", "let me reconsider"
        ])
        if suspicious:
            # Log for review; escalate or retry
            log_for_review(thinking_text, response.content[-1].text)
    elif block.type == "text":
        final_answer = block.text

The judgment call: when to use which approach

The section above applies if you've already decided to use a reasoning model. The prior question — whether you should — is covered in the cost and latency decision tree for reasoning models. But a brief framing here:

Prompting a reasoning model well means understanding that you're paying for search. The model searches through reasoning paths before committing. That search is worth paying for when the space of possible approaches is large and the difference between a mediocre approach and an excellent one is significant. It's not worth paying for when there's one obvious approach and the task is execution.

If a competent human would answer your question in under 30 seconds without deliberation — extracting a date from a document, classifying sentiment, summarizing a paragraph — a reasoning model is the wrong tool regardless of how you prompt it. Route those tasks to a fast standard model or a model cascade.

If a competent human would want an hour to think carefully — designing a system architecture, evaluating a complex legal clause, debugging a subtle concurrency bug — a reasoning model with adequate budget and a clean constraint-focused prompt is probably worth it.

The prompting philosophy follows from this directly. You're not coaxing a standard model into reasoning it wouldn't do on its own. You're defining the problem clearly enough that the model's existing reasoning capability has a target to work toward. Clear problem, hard constraints, precise output format. Everything else is the model's job.

One last thing: the models improve fast, and so does the API surface. The reasoning_effort parameter, the budget_tokens range, the developer role — these are current as of mid-2026 but will change. The underlying principle — short prompts, explicit constraints, no scaffolding — is more stable than any specific parameter. Build your prompting strategy around the mechanism, not the current API surface. Check the official Anthropic extended thinking and OpenAI reasoning documentation for parameter updates, and run your own evals on your own task distribution when specifications drift. For where prompting stops and context construction starts, see from prompt engineering to context engineering.