From Prompt Engineering to Context Engineering: What Actually Changed

The bill arrived in December of last year. A startup's autonomous research agent had been running 24/7 for a month — retrieval, summarization, web calls, report generation. The token costs were three times the projection. When they pulled the traces, the cause was obvious: the agent's system prompt included the current UTC timestamp, formatted to the minute. Every single request hit the cache cold. Fifty thousand requests a day, each paying full input-token price for a 4,000-token system prompt. Removing eight characters from the prompt cut their bill by 70%. That's context engineering, or rather, the cost of ignoring it.

What prompt engineering actually is — and where it stops

Prompt engineering emerged as a practice around 2021-2022 when it became clear that the wording of instructions to a language model produced wildly different outputs. Write "classify the sentiment" and you get one behavior. Write "classify the sentiment as exactly one of: positive, negative, neutral. Output only the label, nothing else" and you get something you can actually parse. The core insight was that models are instruction-following systems, and following instructions well requires clear instructions.

That insight is still correct. For a single-turn classifier, a document summarizer, a simple chatbot, the bottleneck really is the quality of the instruction text. Most of the classic prompt engineering techniques — zero-shot templates, few-shot examples, chain-of-thought elicitation, persona setting, output format specification — operate on the content of what you write and have almost nothing to say about the structure of what surrounds it.

The problem shows up the moment you add a second step. And a third. And retrieval. And tools. And memory.

A production RAG pipeline has no single "prompt." Each query assembles a context from: a system prompt that may be hundreds to thousands of tokens, a set of retrieved document chunks selected by a vector search, any conversation history from prior turns, tool schema definitions if the LLM can call functions, and the user's current question. The model sees all of this together as one flat sequence of tokens. Whether the retrieved chunks are relevant, whether the history has grown stale or contradictory, whether the tool schemas are eating 3,000 tokens the model doesn't need right now — none of that is a prompt wording problem. It's a context architecture problem.

The framing that named it

In June 2025, Andrej Karpathy threw his weight behind a name for the problem in a widely-circulated note: context engineering. The term was already in the air — Shopify's Tobi Lütke had posted the framing days earlier — but Karpathy's endorsement and elaboration made it stick. His analogy: the LLM is the CPU, the context window is RAM, and the context engineer's job is to act like an operating system deciding what to load into memory. The analogy is more precise than it might look.

RAM is finite and shared. Loading the wrong program into memory means the right program doesn't fit. What fits in memory at any moment determines what computations the CPU can perform — you can't process data that isn't loaded. Eviction happens when you run out of space, and what gets evicted is what the system can no longer reference. A CPU with infinite RAM would not need an OS managing memory; a model with an infinite context window would not need context engineering. Neither exists.

The framing also clarifies the relationship between the two disciplines: prompt engineering is like writing good machine instructions; context engineering is like designing the memory layout and OS scheduling policy. Both matter. But once your software (your LLM pipeline) is complex enough, the OS layer dominates.

The four operations

LangChain's 2025 context engineering framework codified four operations that cover the design space. They're worth understanding concretely:

Write means populating state that future model calls will read. A scratchpad where the agent records intermediate conclusions. A structured memory store that persists facts across sessions. A working document the model updates as it makes progress on a multi-step task. Writing context is the discipline of deciding what to preserve and in what form.

Select means retrieving the right subset of available information rather than injecting everything. This is where RAG lives, but the principle extends further: selecting which tools to include in the schema (not all 40, just the 5 relevant to this sub-task), selecting which memory entries are relevant to the current query, selecting which history turns to include and which to drop. LangChain's internal research found that RAG-based tool selection — retrieving only the tools relevant to the current task — produced roughly 3× better accuracy compared to listing every available tool in the schema. The model's effective reasoning space is larger when the irrelevant noise is absent.

Compress means reducing context that has grown too large. Tool outputs are often verbose. Conversation history accumulates. Retrieved documents may overlap. Compress operations include: recursive summarization at turn boundaries, trimming the oldest history once a sliding window fills, post-processing verbose API responses to extract only the fields the model actually needs. The goal is to preserve the information content while reducing the token count.

Isolate means running a sub-agent or sub-task in its own scoped context so it doesn't see state it doesn't need — and can't contaminate the parent's context with irrelevant outputs. A code execution sub-agent only needs the code and the function spec; it doesn't need six prior turns of conversation about user preferences. Isolation is also the right architectural response when two tasks have different context requirements: give them separate windows.

flowchart TD
    TASK[Task / Query]
    
    TASK --> SEL[Select\nwhich docs, tools, memory\nare relevant right now]
    TASK --> WRITE[Write\npopulate scratchpad\nand state for later]

    SEL --> WINDOW[Context Window\nassembled per-request]
    WRITE --> WINDOW

    WINDOW --> MODEL[LLM Call]
    MODEL --> OUTPUT[Output / Tool Calls]

    OUTPUT --> COMP[Compress\ntrim or summarize\nverbose outputs]
    OUTPUT --> ISO[Isolate\nlaunch sub-agent\nwith scoped context]

    COMP --> WINDOW
    ISO --> SUBWIN[Sub-agent\nContext Window]
    SUBWIN --> SUBMODEL[Sub-agent\nLLM Call]
    SUBMODEL --> WINDOW

    style WINDOW fill:#0e7490,color:#fff
    style SUBWIN fill:#4f46e5,color:#fff
    style MODEL fill:#0e7490,color:#fff
    style SUBMODEL fill:#0e7490,color:#fff

Static templates vs. dynamic context assembly

The architectural shift the four operations produce is from static templates to dynamic per-request context assembly. This is worth making concrete.

A static template looks like:

SYSTEM_PROMPT = """
You are a helpful customer support assistant for Acme Corp.
Always be polite. Never discuss competitors.
If you don't know the answer, say so.
"""

def respond(user_message: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-4-5",
        system=SYSTEM_PROMPT,
        messages=[{"role": "user", "content": user_message}]
    )
    return response.content[0].text

This is prompt engineering. The system prompt is a fixed string. Every call sees the same context. For a simple assistant, this is the right level of complexity.

Dynamic context assembly looks like:

def respond(user_message: str, session_id: str) -> str:
    # Select: retrieve relevant docs and only the tools needed
    relevant_chunks = retriever.search(user_message, top_k=3)
    active_tools = tool_selector.get_relevant(user_message, max_tools=5)
    
    # Select: get compressed recent history
    history = memory.get_recent_turns(session_id, max_tokens=2000)
    
    # Select: read the persistent state a prior turn wrote for this session
    session_state = state_store.get(session_id)

    # Assemble context in stable-first order (for caching)
    system = build_system_prompt(
        base_instructions=BASE_INSTRUCTIONS,  # stable prefix — cache hit
        session_state=session_state,           # session-level stable block
    )
    
    user_turn = build_user_turn(
        chunks=relevant_chunks,
        message=user_message,
        # dynamic content comes LAST, after stable prefix
    )
    
    response = client.messages.create(
        model="claude-sonnet-4-5",
        system=system,
        messages=[*history, {"role": "user", "content": user_turn}],
        tools=active_tools,
    )
    
    # Write: persist updated state for the next turn to read
    state_store.put(session_id, update_state(session_state, response))

    # Compress: if history is getting long, trigger summarization
    if memory.token_count(session_id) > 8000:
        memory.compress(session_id)
    
    return response.content[0].text

The second version has explicit context engineering decisions at every step. The instructions themselves (BASE_INSTRUCTIONS) haven't changed. What changed is the orchestration of what surrounds them.

The caching dimension

Prompt caching is where context engineering directly hits the cost line — see Prompt Caching: Cutting Costs 50–90% Without Changing Model Behavior for the full mechanics. The condensed version:

Both Anthropic and OpenAI cache the prefix of the context and charge a fraction of the normal input-token price for cache hits. Anthropic requires explicit cache_control markers; OpenAI applies caching automatically. Both require a minimum of 1,024 tokens in the stable prefix to qualify. The cost reduction is real: 50% from OpenAI, up to 90% on cache reads from Anthropic via cache_control breakpoints (cache writes cost a premium over the base input price).

The structural rule is strict: stable content comes first, dynamic content comes last. System instructions → tool schemas → retrieved reference documents (if they're the same across requests) → then the user's message. This ordering lets the cache boundary fall as early in the token stream as possible.

What breaks the cache silently:

• A timestamp or date in the system prompt ("Today is {{date}}")
• A user ID or session token embedded in system instructions
• Rotating disclaimers that change per deployment
• Tool schemas that shuffle order between requests

{ "type": "prompt-cache", "scenario": "agent", "title": "Cache hit vs miss: where dynamic content lands" }

The cost difference between a cache-hitting agent and a cache-missing one compounds with scale. At 10,000 requests/day with a 4,000-token system prompt at a typical frontier model price ($3/MTok input), the cache-missing version costs $120/day just for the repeated system prompt tokens. The cache-hitting version costs $12/day (Anthropic's ~90% cache-read discount) to $60/day (OpenAI's 50%). Over a 30-day month, that's $3,600 vs $360–$1,800. The prompt text is identical in both cases; the only change is where the dynamic content was placed.

What changes with reasoning models

Reasoning models — o3, Claude Sonnet with extended thinking enabled, Gemini 2.x with thinking mode — change the context engineering picture in two ways.

First, chain-of-thought prompting becomes redundant and sometimes counterproductive. "Think step by step" works on standard models because it elicits intermediate reasoning tokens in the visible output, which the model then conditions on. Reasoning models generate internal "thinking" tokens before the visible response — the step-by-step reasoning happens regardless of whether you ask for it. Adding a CoT instruction doesn't hurt in a minor way; in some tasks it actively constrains the model to a linear reasoning pattern when multi-path or tree-based reasoning would perform better. The 2025-2026 guidance for reasoning models is: state goals and constraints clearly, let the model decide how to reason about them.

Second, reasoning models are significantly more expensive per call, and they spend that budget on thinking tokens. Context engineering decisions about what to include in the context become more important, not less, because each irrelevant token wastes thinking budget as well as context space. See Few-Shot Prompting, Chain-of-Thought, and When Reasoning Models Flip the Script for detailed guidance on what changes per technique.

flowchart LR
    subgraph Standard["Standard Model (GPT-4o, Claude Sonnet)"]
        SP1["System Prompt\n+ 'think step by step'"] --> MOD1[Model]
        MOD1 --> REASON1["Reasoning in\nvisible output"]
        REASON1 --> ANS1[Answer]
    end

    subgraph Reasoning["Reasoning Model (o3, Claude extended thinking)"]
        SP2["System Prompt\n+ goals + constraints"] --> MOD2[Model]
        MOD2 --> THINK["Internal thinking\n(not in output)"]
        THINK --> ANS2[Answer]
    end

    style REASON1 fill:#ffaa00,color:#0a0a0f
    style THINK fill:#4f46e5,color:#fff
    style ANS1 fill:#15803d,color:#fff
    style ANS2 fill:#15803d,color:#fff

Context window as the resource to engineer

The context window visualizer makes the saturation problem concrete:

{ "type": "context-window", "window": 128000, "title": "128k window under agentic load" }

Drag the sliders to simulate an agent under load: system prompt and tools eat 5–8% of the budget at baseline, retrieved chunks another 3–5%. Now push history and tool outputs up to what 20 uncompressed turns would accumulate and watch the bar overflow — the available space for new generation shrinks dramatically, and the model starts losing access to early context because the "lost in the middle" effect means early-position content is attended to less reliably than recent content.

The lost-in-the-middle problem compounds the saturation problem. A model with a 128k context window doesn't have equal access to all 128k positions. Research has consistently found a U-shaped accuracy curve: information at the very beginning and very end of the context is recalled more reliably than information in the middle. See Context Window Management for Agents: Select, Compress, Isolate for the mitigation strategies. The point for this article: context length is not a substitute for context quality.

What breaks in production

Timestamp in the system prompt. Already covered above, but the canonical error pattern: "Today is {datetime.now()}" placed anywhere before the user turn. The cache treats the entire prefix up to that point as a distinct key. Every request is a cache miss. This pattern ships to production because it looks reasonable — why wouldn't you tell the model the current date? — and the cost impact is invisible without explicit cost-per-request tracing.

Instruction-stacking in a single context. A prompt that asks the model to (1) classify intent, (2) extract entities, (3) generate a response, and (4) assess response quality in one call does all four tasks worse than four separate calls, each with a focused context. Attention and context capacity are shared; more tasks in the context means less budget for each. The failure mode is subtle: each task produces output that looks plausible but is less accurate than a focused call, and it's hard to attribute the quality degradation to the context architecture rather than to the model.

Accumulating tool outputs without compression. An agent that appends raw tool output to the context without trimming or summarizing will saturate a 128k window within 30–50 turns if tool outputs are even moderately verbose. The model's behavior changes gradually: it starts omitting steps it remembers seeing earlier, contradicts conclusions it reached before the window began, and eventually truncates. This is diagnosable by tracing the full context at each step — tools like LangSmith and Braintrust make this visible.

Retrieving everything instead of selecting. A RAG pipeline that returns the top-20 chunks instead of top-3 to "be safe" does not necessarily improve answer quality. Irrelevant chunks dilute the context, the relevant signal competes with noise, and the model's effective attention on the right chunk decreases. The right retrieval strategy is a hybrid search with a reranker and a confidence threshold, not a higher top_k.

No versioning on context structure changes. Changing where dynamic content appears in the context (which breaks caching), changing which tool schemas are included (which changes tool-selection behavior), or changing how history is compressed are all prompt-behavior changes in practice. Without a fixed eval suite and without treating these as versioned artifacts, teams can't distinguish regression from improvement. The Prompt Versioning and Evaluation article covers the tooling.

When prompt engineering is sufficient — and when it isn't

This isn't an argument that prompt engineering is obsolete. It's an argument about where the bottleneck is.

Prompt engineering is the right tool when:

• You're making single-turn calls with no retrieval or tools
• The context is fully static across requests (same system prompt, same structure)
• Your main problem is output quality on a well-defined task
• You're working with reasoning models where instruction clarity matters more than context architecture

Context engineering becomes necessary when:

• Your pipeline makes multiple LLM calls that share or build on state
• You're using retrieval and need to decide what to retrieve and how much
• Tool outputs accumulate across turns
• You have memory that needs to persist across sessions
• You're running sub-agents and need to scope their context
• Caching cost savings are material to your unit economics
• You're approaching context window limits on long tasks

The practical checklist: if your "prompt" is assembled by more than three lines of code, you're doing context engineering whether you've named it that or not. Naming it helps because it points to the right design questions — Select, Compress, Isolate, Write — rather than defaulting to "make the instruction clearer."

The versioning and eval requirement

One thing prompt engineering and context engineering share: you cannot improve what you don't measure. Teams that A/B their prompt changes against a fixed eval suite routinely find that intuition-driven edits — changes an engineer judged to be obvious improvements — regress production accuracy about as often as they help. The same dynamic applies to context changes.

Every change to what goes into the context — a new retrieval strategy, a different compression policy, a reordered system prompt — is a behavior change. It needs an eval before shipping. The eval doesn't need to be sophisticated: a fixed set of representative inputs with graded expected outputs, run automatically on every change, with a threshold gate that blocks deployment if accuracy drops. Prompt Versioning, Evaluation, and LLMOps and the system prompt design patterns article cover this in detail.

What the eval catches: silent regressions when context ordering changes break a caching assumption that coincidentally also affected model behavior; retrieval threshold changes that cut relevant chunks; compression policies that lose information the model was using to stay on task.

The decision in practice

Context engineering is not a separate system you build — it's a lens you apply to every architectural decision about what goes into each LLM call.

When designing a new pipeline, walk through each call and ask: What does this model call actually need to see? What can be retrieved just-in-time instead of always present? What from the previous step can be compressed before being passed forward? Should this sub-task get its own isolated context?

When diagnosing a regression, look at the full assembled context before looking at the prompt text. Add tracing that captures the complete input to each LLM call — not just the instruction, but all the chunks, all the history, all the tool outputs. What the model actually sees is usually more informative than what you intended to send it.

When optimizing costs, fix the context structure before negotiating model tiers. Moving a timestamp out of the system prompt, restructuring retrieval to produce a stable prefix, adding compression at conversation checkpoints — these changes routinely produce 50–80% cost reductions with no change to output quality. Switching from a frontier model to a cheaper tier produces 50–80% cost reductions with a quality trade-off you have to manage.

The hierarchy: get the context architecture right first. Then tune the instructions. Then choose the model. In that order.

The next step from here is Context Window Management for Agents for the compression and isolation mechanics, Prompt Caching: Cutting Costs 50–90% for the caching implementation details, and Prompt Anti-Patterns for the specific failure modes that ship silently to production.