# From Prompt Engineering to Context Engineering: What Actually Changed

> Why writing clever prompts is the wrong mental model for 2025 LLM systems, and how the context-window-as-RAM framing changes what you build.

Canonical: https://ironclad.academy/ai-engineering/articles/prompt-to-context-engineering | Difficulty: beginner | Topics: Prompting, Context Engineering, Agents, LLMOps

## Key takeaways
- Context engineering is the discipline of deciding what the model sees at each step — not just what the instruction says.
- The context window is RAM: every token you load displaces something else, and what you load determines what the model can reason about.
- Dynamic content (timestamps, user IDs) placed in the system prompt silently destroys caching and can inflate costs 2–10x versus a cache-stable layout, depending on provider.
- In agent systems, the bottleneck shifts from prompt wording to context architecture — which documents, tools, and history reach each model call.
- Reasoning models invert the chain-of-thought playbook: adding "think step by step" to o3 or Claude extended thinking is working from a 2023 playbook.
- Ship no prompt change without a fixed eval suite; intuition-driven "improvements" measurably degrade production accuracy as often as they help.

> **In one line:** "Prompt engineering" means writing better instructions; "context engineering" means architecting what the model sees at every step of a pipeline — and the second skill is what actually limits production LLM systems in 2025.

When Andrej Karpathy endorsed the term "context engineering" in June 2025 — a phrase already circulating, most visibly from Tobi Lütke — he wasn't announcing a framework. He was popularizing a name for something engineers had already been struggling with without a shared vocabulary. The insight: an LLM is the CPU, the context window is RAM, and your job is to act like an operating system deciding what to load into memory at each moment. That framing makes the problem concrete. RAM is finite, loading the wrong things wastes it, loading too much causes eviction or truncation, and what fits in memory determines what computations are possible.

The shift from prompt engineering to context engineering tracks the shift from single-turn chatbots to multi-step agent systems. A well-tuned system prompt is still the right tool for a classifier or a single-turn assistant. The moment your system makes multiple LLM calls, retrieves documents, executes tools, or orchestrates sub-agents, you have a context architecture problem — and the text of the instructions becomes the least important thing to optimize.

**Key points.**
- Prompt engineering is about *what the instruction says*; context engineering is about *what the model sees* — documents, tool schemas, history, memory, and state that surround the instruction.
- The four core operations: **Write** (populate scratchpads and memory), **Select** (retrieve the right subset, not everything), **Compress** (summarize or prune grown context), **Isolate** (sub-agents with scoped context).
- Static templates work for simple, single-turn systems. Dynamic per-request context assembly is required once retrieval, tools, or multi-step state enter the picture.
- Prompt caching — placing static content first, dynamic content last — can reduce API costs 50–90% with no change to model behavior.
- Reasoning models (o3, Claude extended thinking, Gemini 2.x with thinking) invert the chain-of-thought playbook. Procedural instructions to "think step by step" are redundant on models that already think internally.

**Back-of-the-envelope: context window budget for an agent turn.**
```
Assume a 128k token context window, mid-size task.

System prompt (with tool schemas):     ~2,000 tokens
Retrieved documents (3-5 chunks):      ~3,000 tokens
Conversation history (10 turns):       ~4,000 tokens
Tool call outputs (last 3 calls):      ~6,000 tokens
Working scratchpad / structured state: ~2,000 tokens
─────────────────────────────────────────────────────
Consumed before first new response:   ~17,000 tokens
Available for new generation:        ~111,000 tokens

BUT: if tool outputs accumulate across 20+ turns without compression:
  20 tool outputs × 1,000 tokens avg = 20,000 tokens just in outputs
  Growing history unreduced: +40,000 tokens over 40 turns
→ Window saturates around turn 30-40; model starts losing early context
→ Fix: compress at checkpoints, isolate sub-agent contexts
```

**The anatomy of a context window.**

```mermaid
flowchart LR
    SP["System Prompt\n(instructions, persona, rules)"]
    TOOLS["Tool Schemas\n(JSON definitions)"]
    DOCS["Retrieved Docs\n(RAG chunks)"]
    HIST["Conversation History\n(prior turns)"]
    STATE["Tool Outputs / State\n(results from prior calls)"]
    OUT["Output Reserve\n(max_tokens budget)"]

    SP --> WIN[Context Window]
    TOOLS --> WIN
    DOCS --> WIN
    HIST --> WIN
    STATE --> WIN
    WIN --> OUT

    style WIN fill:#0e7490,color:#fff
    style OUT fill:#15803d,color:#fff
```

**Key design decisions.**

| Decision | Prompt Engineering answer | Context Engineering answer |
|---|---|---|
| What goes in the system prompt? | The task instruction | Stable instructions only; dynamic state goes in later turns |
| How much history to include? | All of it | Sliding window, summaries at checkpoints, or external memory |
| When to add retrieved docs? | Always | Only when retrieval score clears a threshold; otherwise you add noise |
| How to handle tool outputs? | Append raw output | Trim or summarize verbose outputs before they accumulate |
| Sub-task delegation? | One big prompt | Sub-agent with isolated, scoped context window |
| What changes with reasoning models? | Add "think step by step" | Remove CoT instructions; state goals and constraints only |

**If you have 60 seconds, say this.** "Prompt engineering is what I do when I need a better instruction. Context engineering is what I do when I need to control what the model sees — which documents, which tool outputs, which history, in what order, at each step of a pipeline. The context window is finite. Loading the wrong things wastes space and degrades outputs. The discipline is: write state the model will need later, select only what's relevant now, compress what's grown too large, and isolate sub-agents so they don't contaminate each other's context. That's the shift that happened in 2025."



The bill arrived in December of last year. A startup's autonomous research agent had been running 24/7 for a month — retrieval, summarization, web calls, report generation. The token costs were three times the projection. When they pulled the traces, the cause was obvious: the agent's system prompt included the current UTC timestamp, formatted to the minute. Every single request hit the cache cold. Fifty thousand requests a day, each paying full input-token price for a 4,000-token system prompt. Removing eight characters from the prompt cut their bill by 70%. That's context engineering, or rather, the cost of ignoring it.

## What prompt engineering actually is — and where it stops

Prompt engineering emerged as a practice around 2021-2022 when it became clear that the *wording* of instructions to a language model produced wildly different outputs. Write "classify the sentiment" and you get one behavior. Write "classify the sentiment as exactly one of: positive, negative, neutral. Output only the label, nothing else" and you get something you can actually parse. The core insight was that models are instruction-following systems, and following instructions well requires clear instructions.

That insight is still correct. For a single-turn classifier, a document summarizer, a simple chatbot, the bottleneck really is the quality of the instruction text. Most of the classic prompt engineering techniques — zero-shot templates, few-shot examples, chain-of-thought elicitation, persona setting, output format specification — operate on the *content* of what you write and have almost nothing to say about the *structure* of what surrounds it.

The problem shows up the moment you add a second step. And a third. And retrieval. And tools. And memory.

A production RAG pipeline has no single "prompt." Each query assembles a context from: a system prompt that may be hundreds to thousands of tokens, a set of retrieved document chunks selected by a vector search, any conversation history from prior turns, tool schema definitions if the LLM can call functions, and the user's current question. The model sees all of this together as one flat sequence of tokens. Whether the retrieved chunks are relevant, whether the history has grown stale or contradictory, whether the tool schemas are eating 3,000 tokens the model doesn't need right now — none of that is a prompt wording problem. It's a context architecture problem.

## The framing that named it

In June 2025, Andrej Karpathy threw his weight behind a name for the problem in a widely-circulated note: **context engineering**. The term was already in the air — Shopify's Tobi Lütke had posted the framing days earlier — but Karpathy's endorsement and elaboration made it stick. His analogy: the LLM is the CPU, the context window is RAM, and the context engineer's job is to act like an operating system deciding what to load into memory. The analogy is more precise than it might look.

RAM is finite and shared. Loading the wrong program into memory means the right program doesn't fit. What fits in memory at any moment determines what computations the CPU can perform — you can't process data that isn't loaded. Eviction happens when you run out of space, and what gets evicted is what the system can no longer reference. A CPU with infinite RAM would not need an OS managing memory; a model with an infinite context window would not need context engineering. Neither exists.

The framing also clarifies the relationship between the two disciplines: prompt engineering is like writing good machine instructions; context engineering is like designing the memory layout and OS scheduling policy. Both matter. But once your software (your LLM pipeline) is complex enough, the OS layer dominates.

## The four operations

[LangChain's 2025 context engineering framework](https://www.langchain.com/blog/context-engineering-for-agents) codified four operations that cover the design space. They're worth understanding concretely:

**Write** means populating state that future model calls will read. A scratchpad where the agent records intermediate conclusions. A structured memory store that persists facts across sessions. A working document the model updates as it makes progress on a multi-step task. Writing context is the discipline of deciding *what to preserve* and *in what form*.

**Select** means retrieving the right subset of available information rather than injecting everything. This is where RAG lives, but the principle extends further: selecting which tools to include in the schema (not all 40, just the 5 relevant to this sub-task), selecting which memory entries are relevant to the current query, selecting which history turns to include and which to drop. LangChain's internal research found that RAG-based tool selection — retrieving only the tools relevant to the current task — produced roughly 3× better accuracy compared to listing every available tool in the schema. The model's effective reasoning space is larger when the irrelevant noise is absent.

**Compress** means reducing context that has grown too large. Tool outputs are often verbose. Conversation history accumulates. Retrieved documents may overlap. Compress operations include: recursive summarization at turn boundaries, trimming the oldest history once a sliding window fills, post-processing verbose API responses to extract only the fields the model actually needs. The goal is to preserve the information content while reducing the token count.

**Isolate** means running a sub-agent or sub-task in its own scoped context so it doesn't see state it doesn't need — and can't contaminate the parent's context with irrelevant outputs. A code execution sub-agent only needs the code and the function spec; it doesn't need six prior turns of conversation about user preferences. Isolation is also the right architectural response when two tasks have different context requirements: give them separate windows.

```mermaid
flowchart TD
    TASK[Task / Query]
    
    TASK --> SEL[Select\nwhich docs, tools, memory\nare relevant right now]
    TASK --> WRITE[Write\npopulate scratchpad\nand state for later]

    SEL --> WINDOW[Context Window\nassembled per-request]
    WRITE --> WINDOW

    WINDOW --> MODEL[LLM Call]
    MODEL --> OUTPUT[Output / Tool Calls]

    OUTPUT --> COMP[Compress\ntrim or summarize\nverbose outputs]
    OUTPUT --> ISO[Isolate\nlaunch sub-agent\nwith scoped context]

    COMP --> WINDOW
    ISO --> SUBWIN[Sub-agent\nContext Window]
    SUBWIN --> SUBMODEL[Sub-agent\nLLM Call]
    SUBMODEL --> WINDOW

    style WINDOW fill:#0e7490,color:#fff
    style SUBWIN fill:#4f46e5,color:#fff
    style MODEL fill:#0e7490,color:#fff
    style SUBMODEL fill:#0e7490,color:#fff
```

## Static templates vs. dynamic context assembly

The architectural shift the four operations produce is from static templates to dynamic per-request context assembly. This is worth making concrete.

A static template looks like:

```python
SYSTEM_PROMPT = """
You are a helpful customer support assistant for Acme Corp.
Always be polite. Never discuss competitors.
If you don't know the answer, say so.
"""

def respond(user_message: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-4-5",
        system=SYSTEM_PROMPT,
        messages=[{"role": "user", "content": user_message}]
    )
    return response.content[0].text
```

This is prompt engineering. The system prompt is a fixed string. Every call sees the same context. For a simple assistant, this is the right level of complexity.

Dynamic context assembly looks like:

```python
def respond(user_message: str, session_id: str) -> str:
    # Select: retrieve relevant docs and only the tools needed
    relevant_chunks = retriever.search(user_message, top_k=3)
    active_tools = tool_selector.get_relevant(user_message, max_tools=5)
    
    # Select: get compressed recent history
    history = memory.get_recent_turns(session_id, max_tokens=2000)
    
    # Select: read the persistent state a prior turn wrote for this session
    session_state = state_store.get(session_id)

    # Assemble context in stable-first order (for caching)
    system = build_system_prompt(
        base_instructions=BASE_INSTRUCTIONS,  # stable prefix — cache hit
        session_state=session_state,           # session-level stable block
    )
    
    user_turn = build_user_turn(
        chunks=relevant_chunks,
        message=user_message,
        # dynamic content comes LAST, after stable prefix
    )
    
    response = client.messages.create(
        model="claude-sonnet-4-5",
        system=system,
        messages=[*history, {"role": "user", "content": user_turn}],
        tools=active_tools,
    )
    
    # Write: persist updated state for the next turn to read
    state_store.put(session_id, update_state(session_state, response))

    # Compress: if history is getting long, trigger summarization
    if memory.token_count(session_id) > 8000:
        memory.compress(session_id)
    
    return response.content[0].text
```

The second version has explicit context engineering decisions at every step. The instructions themselves (`BASE_INSTRUCTIONS`) haven't changed. What changed is the orchestration of what surrounds them.

## The caching dimension

Prompt caching is where context engineering directly hits the cost line — see [Prompt Caching: Cutting Costs 50–90% Without Changing Model Behavior](/ai-engineering/articles/prompt-caching-cost-optimization) for the full mechanics. The condensed version:

Both Anthropic and OpenAI cache the prefix of the context and charge a fraction of the normal input-token price for cache hits. Anthropic requires explicit `cache_control` markers; OpenAI applies caching automatically. Both require a minimum of 1,024 tokens in the stable prefix to qualify. The cost reduction is real: 50% from OpenAI, up to 90% on cache reads from Anthropic via `cache_control` breakpoints (cache writes cost a premium over the base input price).

The structural rule is strict: **stable content comes first, dynamic content comes last**. System instructions → tool schemas → retrieved reference documents (if they're the same across requests) → then the user's message. This ordering lets the cache boundary fall as early in the token stream as possible.

What breaks the cache silently:
- A timestamp or date in the system prompt (`"Today is {{date}}"`)
- A user ID or session token embedded in system instructions
- Rotating disclaimers that change per deployment
- Tool schemas that shuffle order between requests

[Interactive visualizer on the original page: prompt-cache — Cache hit vs miss: where dynamic content lands]

The cost difference between a cache-hitting agent and a cache-missing one compounds with scale. At 10,000 requests/day with a 4,000-token system prompt at a typical frontier model price ($3/MTok input), the cache-missing version costs $120/day just for the repeated system prompt tokens. The cache-hitting version costs $12/day (Anthropic's ~90% cache-read discount) to $60/day (OpenAI's 50%). Over a 30-day month, that's $3,600 vs $360–$1,800. The prompt text is identical in both cases; the only change is where the dynamic content was placed.

## What changes with reasoning models

Reasoning models — o3, Claude Sonnet with extended thinking enabled, Gemini 2.x with thinking mode — change the context engineering picture in two ways.

First, chain-of-thought prompting becomes redundant and sometimes counterproductive. "Think step by step" works on standard models because it elicits intermediate reasoning tokens in the visible output, which the model then conditions on. Reasoning models generate internal "thinking" tokens before the visible response — the step-by-step reasoning happens regardless of whether you ask for it. Adding a CoT instruction doesn't hurt in a minor way; in some tasks it actively constrains the model to a linear reasoning pattern when multi-path or tree-based reasoning would perform better. The 2025-2026 guidance for reasoning models is: state goals and constraints clearly, let the model decide how to reason about them.

Second, reasoning models are significantly more expensive per call, and they spend that budget on thinking tokens. Context engineering decisions about *what to include in the context* become more important, not less, because each irrelevant token wastes thinking budget as well as context space. See [Few-Shot Prompting, Chain-of-Thought, and When Reasoning Models Flip the Script](/ai-engineering/articles/few-shot-cot-reasoning-models) for detailed guidance on what changes per technique.

```mermaid
flowchart LR
    subgraph Standard["Standard Model (GPT-4o, Claude Sonnet)"]
        SP1["System Prompt\n+ 'think step by step'"] --> MOD1[Model]
        MOD1 --> REASON1["Reasoning in\nvisible output"]
        REASON1 --> ANS1[Answer]
    end

    subgraph Reasoning["Reasoning Model (o3, Claude extended thinking)"]
        SP2["System Prompt\n+ goals + constraints"] --> MOD2[Model]
        MOD2 --> THINK["Internal thinking\n(not in output)"]
        THINK --> ANS2[Answer]
    end

    style REASON1 fill:#ffaa00,color:#0a0a0f
    style THINK fill:#4f46e5,color:#fff
    style ANS1 fill:#15803d,color:#fff
    style ANS2 fill:#15803d,color:#fff
```

## Context window as the resource to engineer

The context window visualizer makes the saturation problem concrete:

[Interactive visualizer on the original page: context-window — 128k window under agentic load]

Drag the sliders to simulate an agent under load: system prompt and tools eat 5–8% of the budget at baseline, retrieved chunks another 3–5%. Now push history and tool outputs up to what 20 uncompressed turns would accumulate and watch the bar overflow — the available space for new generation shrinks dramatically, and the model starts losing access to early context because the "lost in the middle" effect means early-position content is attended to less reliably than recent content.

The lost-in-the-middle problem compounds the saturation problem. A model with a 128k context window doesn't have equal access to all 128k positions. Research has consistently found a U-shaped accuracy curve: information at the very beginning and very end of the context is recalled more reliably than information in the middle. See [Context Window Management for Agents: Select, Compress, Isolate](/ai-engineering/articles/context-window-management-agents) for the mitigation strategies. The point for this article: context length is not a substitute for context quality.

## What breaks in production

**Timestamp in the system prompt.** Already covered above, but the canonical error pattern: `"Today is {datetime.now()}"` placed anywhere before the user turn. The cache treats the entire prefix up to that point as a distinct key. Every request is a cache miss. This pattern ships to production because it looks reasonable — why wouldn't you tell the model the current date? — and the cost impact is invisible without explicit cost-per-request tracing.

**Instruction-stacking in a single context.** A prompt that asks the model to (1) classify intent, (2) extract entities, (3) generate a response, and (4) assess response quality in one call does all four tasks worse than four separate calls, each with a focused context. Attention and context capacity are shared; more tasks in the context means less budget for each. The failure mode is subtle: each task produces output that looks plausible but is less accurate than a focused call, and it's hard to attribute the quality degradation to the context architecture rather than to the model.

**Accumulating tool outputs without compression.** An agent that appends raw tool output to the context without trimming or summarizing will saturate a 128k window within 30–50 turns if tool outputs are even moderately verbose. The model's behavior changes gradually: it starts omitting steps it remembers seeing earlier, contradicts conclusions it reached before the window began, and eventually truncates. This is diagnosable by tracing the full context at each step — tools like [LangSmith and Braintrust](/ai-engineering/articles/llm-observability-tooling-compared) make this visible.

**Retrieving everything instead of selecting.** A RAG pipeline that returns the top-20 chunks instead of top-3 to "be safe" does not necessarily improve answer quality. Irrelevant chunks dilute the context, the relevant signal competes with noise, and the model's effective attention on the right chunk decreases. The right retrieval strategy is a [hybrid search](/ai-engineering/articles/hybrid-search-bm25-dense-reranking) with a reranker and a confidence threshold, not a higher `top_k`.

**No versioning on context structure changes.** Changing where dynamic content appears in the context (which breaks caching), changing which tool schemas are included (which changes tool-selection behavior), or changing how history is compressed are all prompt-behavior changes in practice. Without a fixed eval suite and without treating these as versioned artifacts, teams can't distinguish regression from improvement. The [Prompt Versioning and Evaluation](/ai-engineering/articles/prompt-versioning-evaluation-llmops) article covers the tooling.

## When prompt engineering is sufficient — and when it isn't

This isn't an argument that prompt engineering is obsolete. It's an argument about where the bottleneck is.

**Prompt engineering is the right tool** when:
- You're making single-turn calls with no retrieval or tools
- The context is fully static across requests (same system prompt, same structure)
- Your main problem is output quality on a well-defined task
- You're working with reasoning models where instruction clarity matters more than context architecture

**Context engineering becomes necessary** when:
- Your pipeline makes multiple LLM calls that share or build on state
- You're using retrieval and need to decide *what* to retrieve and *how much*
- Tool outputs accumulate across turns
- You have memory that needs to persist across sessions
- You're running sub-agents and need to scope their context
- Caching cost savings are material to your unit economics
- You're approaching context window limits on long tasks

The practical checklist: if your "prompt" is assembled by more than three lines of code, you're doing context engineering whether you've named it that or not. Naming it helps because it points to the right design questions — Select, Compress, Isolate, Write — rather than defaulting to "make the instruction clearer."

## The versioning and eval requirement

One thing prompt engineering and context engineering share: you cannot improve what you don't measure. Teams that A/B their prompt changes against a fixed eval suite routinely find that intuition-driven edits — changes an engineer judged to be obvious improvements — regress production accuracy about as often as they help. The same dynamic applies to context changes.

Every change to what goes into the context — a new retrieval strategy, a different compression policy, a reordered system prompt — is a behavior change. It needs an eval before shipping. The eval doesn't need to be sophisticated: a fixed set of representative inputs with graded expected outputs, run automatically on every change, with a threshold gate that blocks deployment if accuracy drops. [Prompt Versioning, Evaluation, and LLMOps](/ai-engineering/articles/prompt-versioning-evaluation-llmops) and the [system prompt design patterns](/ai-engineering/articles/system-prompt-design-patterns) article cover this in detail.

What the eval catches: silent regressions when context ordering changes break a caching assumption that coincidentally also affected model behavior; retrieval threshold changes that cut relevant chunks; compression policies that lose information the model was using to stay on task.

## The decision in practice

Context engineering is not a separate system you build — it's a lens you apply to every architectural decision about what goes into each LLM call.

When designing a new pipeline, walk through each call and ask: What does this model call actually need to see? What can be retrieved just-in-time instead of always present? What from the previous step can be compressed before being passed forward? Should this sub-task get its own isolated context?

When diagnosing a regression, look at the full assembled context before looking at the prompt text. Add tracing that captures the complete input to each LLM call — not just the instruction, but all the chunks, all the history, all the tool outputs. What the model actually sees is usually more informative than what you intended to send it.

When optimizing costs, fix the context structure before negotiating model tiers. Moving a timestamp out of the system prompt, restructuring retrieval to produce a stable prefix, adding compression at conversation checkpoints — these changes routinely produce 50–80% cost reductions with no change to output quality. Switching from a frontier model to a cheaper tier produces 50–80% cost reductions with a quality trade-off you have to manage.

The hierarchy: get the context architecture right first. Then tune the instructions. Then choose the model. In that order.

The next step from here is [Context Window Management for Agents](/ai-engineering/articles/context-window-management-agents) for the compression and isolation mechanics, [Prompt Caching: Cutting Costs 50–90%](/ai-engineering/articles/prompt-caching-cost-optimization) for the caching implementation details, and [Prompt Anti-Patterns](/ai-engineering/articles/prompt-anti-patterns) for the specific failure modes that ship silently to production.

## Frequently asked questions
Q: What is context engineering and how does it differ from prompt engineering?
A: Prompt engineering is the practice of crafting the text of instructions to elicit better responses from a model. Context engineering is the broader discipline of deciding what information — system instructions, retrieved documents, tool schemas, memory, conversation history, tool outputs — goes into the context window at each step of a pipeline, and in what order and quantity. The distinction matters most in multi-step agent systems, where no single static prompt exists: the model sees a dynamically assembled window that changes with every turn.

Q: Who coined the term context engineering?
A: The phrase was already circulating — Shopify's Tobi Lütke gets the usual proximate credit — when Andrej Karpathy popularized it in June 2025 with a widely-shared endorsement of "context engineering" over "prompt engineering". His framing — LLM as CPU, context window as RAM, context engineering as the OS deciding what to load — captured the shift from single-turn prompt tuning to multi-step context architecture.

Q: What are the four core context operations in the context engineering framework?
A: LangChain's 2025 framework codified four operations: Write (populating scratchpads, memory stores, and state that later turns will read), Select (retrieving the right subset via RAG or semantic search rather than dumping everything in), Compress (summarizing or pruning context that has grown too large), and Isolate (spinning up sub-agents with scoped, sandboxed context so irrelevant state stays out). Applying these four operations to every stage of a pipeline is what separates context engineering from prompt engineering.

Q: Does context engineering replace prompt engineering?
A: No. For a single-turn chatbot or a simple classification call, a well-crafted system prompt is still the right tool. Context engineering adds a layer on top: once your system makes multiple calls, retrieves documents, uses tools, or runs sub-agents, you need explicit decisions about what each call sees. Think of prompt engineering as writing the instructions for one conversation, and context engineering as architecting what state those instructions are embedded in across a whole workflow.

Q: How much can prompt caching reduce LLM API costs?
A: With Anthropic's explicit cache_control markers, teams report 50–90% cost reduction on repeated calls that share a long stable prefix (system prompt, tools, large document sets). OpenAI provides automatic 50% cost reduction on cached prefixes with no markers required. Both require a minimum prefix of 1,024 tokens to qualify. The catch: any dynamic content — timestamps, user IDs, session tokens — placed before the dynamic turn in the prompt breaks the cache on every request, eliminating the savings entirely.

Q: When should I still use chain-of-thought prompting in 2025-2026?
A: Chain-of-thought ("think step by step") instructions remain useful on standard non-reasoning models like GPT-4o or Claude Sonnet when you need intermediate reasoning visible in the output. On reasoning models — o3, Claude extended thinking, Gemini 2.x with thinking enabled — those instructions are redundant and sometimes counterproductive: the model is already thinking internally. For reasoning models, state your goals and constraints clearly; do not instruct reasoning procedure.
