Few-shot prompting, chain-of-thought, and reasoning models

A team I know shipped a legal document analysis pipeline in late 2024. Standard model, careful chain-of-thought prompting, 20 worked examples in the prompt, 88% accuracy on their eval set. In mid-2025 they upgraded to a reasoning model thinking it would help. Accuracy dropped to 81%. After three weeks of prompt tuning with no improvement they almost rolled back. What they hadn't done was read the docs on extended thinking.

The instruction they kept in their prompt — "Think through this clause carefully. First, identify the key obligation. Second, check for ambiguity. Third, flag any deviation from standard terms." — was telling the model how to reason. The model was already doing that, internally, on its own schedule. Layering an external reasoning scaffold on top of an internal one doesn't combine. It interferes.

Removing those instructions and replacing them with "Your goal is to identify non-standard contract obligations. Flag any clause that deviates from the baseline. If a clause is ambiguous, say so explicitly." got them to 93%.

This article covers everything between zero-shot and that insight: when few-shot examples help, what chain-of-thought actually does, why the CoT trace isn't a trustworthy audit trail, and how to rebuild your prompting instincts for reasoning models.

Zero-shot, few-shot, and many-shot: what you're actually buying

Zero-shot prompting is a description of the task with no examples. For any task that's common in the model's training distribution — summarization, translation, sentiment classification, standard coding patterns — zero-shot works fine. The model has seen enough variations in pretraining and instruction-tuning to know what you want.

Zero-shot breaks down when the output format is unusual, when your domain has specific conventions the model wouldn't know, or when you need a very specific interpretation of an ambiguous instruction. "Classify this customer feedback as positive, negative, or neutral" is clear enough. "Classify this customer feedback as NEEDS_ATTENTION, ESCALATE, ROUTINE, or UNSOLICITED_FEATURE_REQUEST and return JSON" is not — the model has no baseline for what distinguishes NEEDS_ATTENTION from ESCALATE in your specific business context.

Few-shot prompting solves this by showing rather than telling. You include 3–8 examples of input → output pairs before the actual task. The model patterns-matches against them.

# Anthropic SDK, Python
import anthropic

client = anthropic.Anthropic()

SYSTEM = """You classify customer feedback for a B2B SaaS product.
Return JSON: {"label": "<LABEL>", "confidence": <0-1>, "reason": "<one sentence>"}"""

EXAMPLES = [
    {
        "input": "The API rate limits are too low for our use case. We're hitting them constantly.",
        "output": '{"label": "ESCALATE", "confidence": 0.92, "reason": "Customer blocked by a product limit — revenue risk."}'
    },
    {
        "input": "Would be great if the dashboard had a dark mode.",
        "output": '{"label": "UNSOLICITED_FEATURE_REQUEST", "confidence": 0.95, "reason": "Enhancement request, no urgency signal."}'
    },
    {
        "input": "Login is failing for two of my users since yesterday.",
        "output": '{"label": "NEEDS_ATTENTION", "confidence": 0.97, "reason": "Active access issue but not escalation-level yet."}'
    },
]

def build_prompt(user_input: str) -> list[dict]:
    messages = []
    for ex in EXAMPLES:
        messages.append({"role": "user", "content": ex["input"]})
        messages.append({"role": "assistant", "content": ex["output"]})
    messages.append({"role": "user", "content": user_input})
    return messages

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=256,
    system=SYSTEM,
    messages=build_prompt("We can't export to CSV. We need this for compliance audits by Friday."),
)
print(response.content[0].text)
# → {"label": "ESCALATE", "confidence": 0.94, "reason": "Compliance deadline creates hard urgency."}

Notice what the examples are teaching: the distinction between your four labels, the JSON schema, the expected confidence granularity, and the single-sentence reason style. None of that is in the system prompt description. The examples carry the behavioral specification.

Many-shot prompting extends this to dozens or hundreds of examples, now practical with larger context windows (128K–1M tokens). Research shows accuracy continues to improve up to roughly 20–50 examples for classification tasks before plateauing. Beyond that, you're mostly padding the context. The exception is extremely long-tail taxonomies — one study found meaningful gains up to 100+ examples when there were 50+ possible output classes.

But count isn't the variable that matters most. Quality is. Three clear, edge-case-spanning, mutually-consistent examples will outperform twenty examples that overlap heavily and include a couple with ambiguous labels. This is where few-shot pollution causes silent damage: examples that subtly contradict each other, or examples that show a different labeling scheme than your instruction specifies, produce inconsistent outputs that are hard to debug because the model is doing exactly what it was shown, just not what you intended. See Prompt Anti-Patterns for the taxonomy.

Chain-of-thought: what it does and what it costs

Chain-of-thought (CoT) prompting elicits intermediate reasoning steps before the final answer. The canonical form is the zero-shot CoT instruction "Let's think step by step" discovered by Kojima et al. (2022). Few-shot CoT includes worked examples where the example answers show the reasoning trace explicitly.

What CoT buys on standard models is real: 10–30% accuracy improvement on multi-step reasoning tasks — math, code generation, multi-hop question answering, logical deduction. The improvement is largest on tasks that genuinely require intermediate state: "if Alice is taller than Bob, and Bob is taller than Carol, who is shortest?" requires chained inference that a standard model will often shortcut incorrectly without prompting to slow down.

The mechanism is somewhat understood. When a standard model generates text, each token is produced by attending to all previous tokens. If the model writes "First, Alice > Bob; second, Bob > Carol; therefore Carol is shortest" before the final answer, those intermediate statements are in the context window and the final token prediction has explicit ground truth to attend to. The model has effectively written a scratchpad for itself. Without CoT, it tries to compress all that computation into a single forward pass of the prediction head, which fails for sufficiently complex chains.

sequenceDiagram
    participant P as Prompt
    participant M as Standard Model
    participant O as Output

    P->>M: "Who is shortest? Think step by step."
    M->>O: "Step 1: Alice > Bob (given)."
    M->>O: "Step 2: Bob > Carol (given)."
    M->>O: "Step 3: Therefore Carol < Bob < Alice."
    M->>O: "Answer: Carol is shortest."
    Note over O: Each step attends to previous steps

Few-shot CoT takes this further by showing worked examples:

Q: A train travels 300km in 2.5 hours, then 180km in 1.5 hours. What is its average speed?
A: Total distance = 300 + 180 = 480 km.
   Total time = 2.5 + 1.5 = 4 hours.
   Average speed = 480 / 4 = 120 km/h.
   Answer: 120 km/h.

Q: A car uses 8L of fuel per 100km. How much fuel for a 350km trip?
A: [model fills this in]

The model learns both the format and the style of showing work before stating the answer. This matters for your application because it makes the answer more reliable and gives you an intermediate representation to validate.

The cost of CoT

Each CoT step adds tokens — typically 200–600 output tokens for a medium complexity problem. At illustrative 2026 pricing of $15/1M output tokens for a mid-tier frontier model, 500 extra CoT tokens per request costs $0.0075 per call. At 100,000 calls/day that is $750/day in CoT overhead — about $22,500/month. If CoT isn't buying you accuracy on that task, you're paying for the appearance of reasoning.

Worse, CoT output tokens count toward your context in multi-turn conversations. A pipeline that does 5 CoT reasoning steps in sequence accumulates 2,500–3,000 tokens of visible reasoning before the next user turn. In an agent context this gets out of hand fast. See Context Window Management for Agents for how to manage this.

The faithfulness problem

Here is an inconvenient finding from the 2025 literature: the reasoning trace you see in CoT output does not necessarily reflect how the model computed the answer.

Models can produce a plausible-looking derivation of an answer that was actually computed along a different internal path. The extreme version of this is "sycophantic reasoning" — if you show the model a preferred answer, it generates a chain of reasoning that appears to derive that answer, even when it would have reached the opposite answer independently. The reasoning is post-hoc text generation, not a decoded decision tree.

The practical implication: you can catch obvious errors in CoT traces (a miscalculation that the final answer contradicts), but you cannot treat the trace as an auditable proof of the model's decision-making process. The trace is useful for debugging and for catching gross errors. It is not a compliance artifact. If you need true interpretability, you need a different architecture — structured extraction with grounding checks, for instance.

This doesn't make CoT worthless. The scratchpad mechanism still improves accuracy even when the scratchpad is partially post-hoc. But it changes what you can rely on it for.

Self-consistency: the cheap upgrade before you reach for trees

Before you build any branching orchestration, there is a simpler rung on the ladder. Self-consistency (Wang et al., 2022) samples the same CoT prompt k times at temperature above zero and takes a majority vote over the final answers, discarding the reasoning traces. The intuition: a hard problem has many valid reasoning paths to the right answer but the wrong answers are scattered — sampling and voting averages out individual bad chains. On math benchmarks the original paper reported double-digit accuracy gains over single-pass CoT (roughly +11 to +18 points on GSM8K-class tasks), and the technique needs no orchestration logic at all: the k samples are independent, so you fire them in parallel and your latency is one call, not k.

The cost is exactly what you'd expect: k× the output tokens. Take the CoT overhead math from earlier — 500 CoT tokens at $15/1M output is $0.0075 per call. At k=5, that becomes $0.0375 per call, and the $750/day CoT bill at 100K calls/day becomes $3,750/day. Input tokens multiply too unless your provider caches the shared prefix. So self-consistency earns its keep on tasks where single-pass CoT is measurably flaky and the answer space is discrete enough to vote on (a number, a label, a verdict). It does nothing for open-ended generation, where "majority vote" has no meaning.

And the same caveat as CoT itself applies: on reasoning models, self-consistency is mostly redundant. The extended internal deliberation already buys much of the variance reduction you'd get from voting, and some providers sell the residual gap back to you as a premium tier that samples multiple internal traces server-side. If you're tempted to run k=5 against a reasoning model, run the eval first — you'll usually find you're paying five times for one model's worth of improvement.

Tree-of-thought and beyond

Tree-of-thought (ToT) prompting extends CoT by exploring multiple reasoning branches in parallel, evaluating each, and backtracking when branches look unpromising. It maps more closely to how humans actually solve hard problems — not linear deduction but branching hypothesis-test cycles.

flowchart TD
    ROOT[Problem] --> B1[Branch 1\nAssumption A]
    ROOT --> B2[Branch 2\nAssumption B]
    ROOT --> B3[Branch 3\nAssumption C]
    B1 --> E1{Evaluate}
    B2 --> E2{Evaluate}
    B3 --> E3{Evaluate}
    E1 -->|dead end| PRUNE1[Prune]
    E2 -->|promising| B2A[Branch 2a]
    E2 -->|promising| B2B[Branch 2b]
    E3 -->|dead end| PRUNE3[Prune]
    B2A --> FINAL[Best path → answer]

    style PRUNE1 fill:#ff2e88,color:#111
    style PRUNE3 fill:#ff2e88,color:#111
    style FINAL fill:#15803d,color:#fff

The gains are real but lopsided. In the original ToT paper (Yao et al., 2023), GPT-4 solved 74% of Game of 24 puzzles with ToT versus 4% with linear CoT — a 70-point jump, on a narrow combinatorial benchmark built to reward search. On tasks without that branch-and-prune structure, gains shrink to modest or nil. But it costs more: ToT requires multiple LLM calls (generate branches, evaluate, prune, generate deeper), careful orchestration code, and 3–5× the token budget of single-pass CoT. For high-volume, latency-sensitive inference, that overhead is prohibitive.

The honest production recommendation: if a task needs tree-of-thought to get right, route it to a reasoning model instead. To be precise about the mechanism: these models do not run a literal tree search at inference. RL training (RLVR/GRPO) teaches them to explore alternatives, self-check, and backtrack within a single serial chain-of-thought — one long trace that says "wait, that doesn't work, try B" where ToT would spawn a branch. In practice that substitutes for external ToT orchestration on most tasks. You pay per-token for reasoning tokens rather than orchestration complexity.

When reasoning models flip the script

Reasoning models — OpenAI's o3/o4 series, Anthropic's Claude Sonnet and Opus with extended thinking (Claude 3.7 through 4.x), Google's Gemini 2.x with thinking — perform internal chain-of-thought deliberation before emitting their answer. The thinking is trained in, not prompted in.

When you add "think step by step" to a call to one of these models, you are asking it to do something it is already doing. At best, your instruction is redundant. At worst, it interferes — the model may try to emit explicit reasoning text that duplicates its hidden deliberation, spending tokens on a visible trace that serves neither you nor it.

The 2025–2026 guidance from Anthropic, OpenAI, and practitioners who've shipped with these models is consistent: state goals and constraints, not reasoning procedures.

The other lever that matters — and the one that dominates cost and latency in production — is the thinking budget. Every major provider exposes a knob for how much internal deliberation the model spends before answering: Anthropic's budget_tokens on extended thinking, OpenAI's reasoning effort parameter, Gemini's thinking budget. This is not a tuning nicety; it moves reasoning output by an order of magnitude, which means the $102/day figure in the summary block is really a range from roughly $50 to $300+ depending on where you set it. The production pattern is to tier it: cap the budget low (or set effort to low/minimal) for tasks you route to the reasoning model out of caution rather than necessity, and reserve high budgets for the clauses your triage step actually flags as hard. Teams that leave the budget at its default are usually paying for deliberation their easy cases don't need — the same money-for-the-appearance-of-reasoning failure as blanket CoT, one abstraction level up.

# Standard model — explicit CoT helps
standard_prompt = """Analyze this contract clause and determine if it's standard or non-standard.
Think step by step:
1. Identify the key obligation
2. Check for unusual scope or carve-outs
3. Compare to market standard
4. Give your verdict

Clause: {clause_text}"""

# Reasoning model — state the goal, not the procedure
reasoning_prompt = """Analyze this contract clause.
Your goal: determine if the clause is standard or non-standard for a SaaS MSA.
Constraints:
- If the clause limits liability below 12 months of fees, flag as non-standard.
- If the clause includes IP ownership of customer data, flag as non-standard immediately.
- Ambiguous language should default to flagging as non-standard.

Return JSON: {{"verdict": "standard"|"non-standard", "reason": "<one sentence>", "risk_level": "low"|"medium"|"high"}}

Clause: {clause_text}"""

For reasoning models, few-shot examples still add value — but what they teach is different. You're not teaching reasoning procedure. You're teaching output format, edge-case interpretation, and the calibration of your verdict categories.

# Reasoning model with few-shot examples — output structure only
EXAMPLES_FOR_REASONING_MODEL = [
    {
        "clause": "Vendor liability is capped at fees paid in the preceding 30 days.",
        "output": '{"verdict": "non-standard", "reason": "30-day cap is significantly below the 12-month market floor.", "risk_level": "high"}'
    },
    {
        "clause": "Vendor liability is capped at fees paid in the preceding 12 months.",
        "output": '{"verdict": "standard", "reason": "12-month liability cap is at market standard for SaaS MSAs.", "risk_level": "low"}'
    },
]

Notice: no reasoning steps in the example answers. The model doesn't need them. You're calibrating how to translate its internal assessment into your label vocabulary, not teaching it how to think.

A taxonomy of what you're actually teaching

Whether you're prompting a standard or reasoning model, it helps to be explicit about which layer your examples or instructions are targeting:

flowchart TD
    TASK[Prompt contents] --> F[Format layer\nJSON schema, field names, response structure]
    TASK --> D[Domain layer\nCategory definitions, threshold values, business rules]
    TASK --> T[Tone layer\nStyle, length, register, audience]
    TASK --> R[Reasoning layer\nHow to derive the answer — standard model only]

    F --> BOTH[Both model types benefit]
    D --> BOTH
    T --> BOTH
    R --> STD[Standard models only\nSkip for reasoning models]

    style R fill:#00e5ff,color:#0a0a0f
    style STD fill:#00e5ff,color:#0a0a0f
    style BOTH fill:#15803d,color:#fff

This framing makes prompt construction deliberate instead of intuitive. Before including an example or an instruction, ask which layer it belongs to. If it's targeting the reasoning layer and you're on a reasoning model, it's probably overhead.

What breaks

Few-shot pollution. Examples that look correct but subtly contradict the instruction or each other will silently degrade output quality. A set of classification examples where three show ESCALATE for urgent-but-low-revenue feedback and one shows ROUTINE for similar feedback will cause the model to output inconsistently on similar inputs. This is hard to catch without a fixed eval suite because individual outputs look reasonable — it's the distribution that breaks. The fix is to review examples for mutual consistency before shipping, and run a regression against a labeled test set when you add or change examples.

Applying CoT to reasoning models. As described above. Symptoms: slightly worse accuracy than the baseline, higher token cost, longer latency. Often misread as a problem with the model or the task. The diagnostic is simple: strip all procedural instructions from the prompt and run the eval again.

CoT trace as truth. Using the reasoning trace as the authoritative explanation for a model decision in a regulated context — "the model explained its reasoning and it was sound" — is a misuse of CoT. The trace is a text generation artifact correlated with the answer, not a verified decision path. Do not put this in a compliance workflow without additional grounding checks.

Example leakage into parsers. If your few-shot examples contain specific strings and your downstream parser hardcodes to those exact strings, a model upgrade that generates the same answer but formatted slightly differently will silently break your pipeline. Parse structure (JSON fields), not string patterns. This failure mode is covered in depth in Prompt Anti-Patterns.

Too many examples, wrong examples. Adding more examples after diminishing returns bloat context, cost, and latency without accuracy benefit. A 20-example prompt on a 128K-token context window might seem fine until you're in an agent loop where the system prompt is regenerated every turn — that's 10,000 tokens of overhead on each of 10 iterations. See Prompt Caching: Cutting Costs 50-90% for how to cache the example block to amortize this cost.

Ignoring base rates in few-shot selection. If your task has class imbalance — 90% of real inputs are ROUTINE and 10% are ESCALATE — but your examples are 50/50, the model will over-predict the minority class. Match example distribution to expected production distribution, or explicitly state the expected class rates in the system prompt.

A worked numbers block: example token budget

Scenario: 50K API calls/day, feedback classification pipeline.

Zero-shot prompt: 150 tokens system + 80 tokens input = 230 tokens/call
  Cost (input): 50,000 × 230 × $0.000003 = $34.50/day

Few-shot (5 examples, ~200 tokens each): 150 + 1,000 + 80 = 1,230 tokens/call
  Cost (input): 50,000 × 1,230 × $0.000003 = $184.50/day
  Accuracy gain: +12% on classification ambiguity (illustrative)

Few-shot + CoT instruction + visible CoT output (400 tokens):
  Input: 1,330 tokens. Output: 400 CoT + 60 answer = 460 tokens/call
  Cost (in + out): 50,000 × (1,330×$0.000003 + 460×$0.000015) = $199.50 + $345 = $544.50/day
  Accuracy gain: +4% over few-shot alone (diminishing returns on classification tasks)

Few-shot examples cached (Anthropic cache_control, 85% hit rate):
  Cache hit: 1,000 tokens at $0.0000003 (90% discount) + 230 tokens normal
  Effective cost: 50,000 × (85%×(1,000×$0.0000003 + 230×$0.000003) + 15%×1,230×$0.000003)
  ≈ 50,000 × (0.85×$0.00099 + 0.15×$0.00369) = $42.08 + $27.68 = $69.76/day

Verdict: few-shot with caching costs 2× zero-shot, not 5×.
Without caching, few-shot + CoT costs 15× zero-shot.

The takeaway is not to avoid few-shot — it's to cache the example block. Structuring your prompt so the static few-shot examples come before any dynamic content (user input, timestamps, session data) is a prerequisite for this to work. The Prompt Caching article covers the structural rules in detail.

The decision in practice

Given everything above, here is how to make the call:

Use zero-shot when the task is a common pattern (summarize, translate, classify sentiment), format requirements are simple, and you have no examples or they'd be hard to curate cleanly. Run against your eval set before shipping to confirm.

Use few-shot when the output format is non-standard, you have domain-specific label definitions or thresholds that description alone can't convey, tone or style matching matters, or your zero-shot eval is below threshold and you have clean labeled examples to draw from. Keep examples to 3–8, verify mutual consistency, and cache the example block.

Use CoT when the task requires multi-step reasoning on a standard model, you need the intermediate trace for debugging or lightweight validation, or your eval shows a meaningful accuracy gap that other techniques haven't closed. Skip it on reasoning models.

Use tree-of-thought when the task involves hard planning or combinatorial search, you have latency budget for multiple API calls, and accuracy on hard cases is the dominant constraint. In most cases, routing to a reasoning model is simpler. For the economics of that routing decision, see Cost, Latency, and the Model-Selection Decision Tree for Reasoning.

Use reasoning models when accuracy on hard multi-step tasks is the primary goal and you can absorb the cost, or when you're currently spending heavily on CoT orchestration that a reasoning model would replace with a single clean call. Rebuild your prompts: goals and constraints instead of procedure, example outputs instead of example reasoning traces. Your existing prompts will not transfer without revision.

The broader pattern: these techniques are not a hierarchy where each level is always better. They are tools for different problems. A well-curated 3-example few-shot prompt on a standard model will beat a poorly constructed reasoning-model prompt on the same task. The model is not the variable that moves the needle; the prompt architecture is — which is what From Prompt Engineering to Context Engineering is ultimately about.

One thing the 2025 research makes plain: the team that iterates with a fixed eval suite finds out when a prompt change helped or hurt. The team that ships by intuition finds out in production.