Context Engineering: Filling the Window Right

Two contexts. Same model. Same question: "Does our refund policy cover digital purchases?"

Context A arrives with a well-structured system prompt, the current policy document at position one, and a clean user turn. The model answers correctly.

Context B has the same system prompt, the same policy document — but it's buried under six retrieved chunks of vaguely related legal boilerplate, a session timestamp injected eight months ago by a developer who thought it might help, and a 14-turn conversation history that's never been pruned. The model answers confidently. And wrong.

Nothing broke. No exception was thrown. No metric turned red. The context just filled up badly, and the model made the best of it.

That's the problem this module is about.

The mental model: your context window is RAM

Andrej Karpathy named this frame clearly in mid-2025: the LLM is the CPU, the context window is RAM, and your job as an AI engineer is the operating system deciding what to load into that RAM before the processor sees it. The analogy is imprecise in some ways but useful in one specific way: RAM is finite, what's loaded in it determines what the CPU can compute, and loading the wrong things — or loading them in the wrong order — produces wrong answers even from a correct processor.

Prompt engineering is about the quality of a single instruction: can you phrase this task so the model does it right? That skill matters. But in any real production system — a chatbot with a knowledge base, an agent calling tools, a pipeline retrieving documents — the bottleneck isn't usually your phrasing. It's the assembly of everything in that window.

Context engineering is the broader discipline: write, select, compress, and isolate what goes into the window across the system prompt, tool schemas, retrieved documents, conversation history, and agent state. Phrasing is maybe 10% of the work.

flowchart TD
    A["System prompt<br/>(static)"] --> W["Context Window<br/>128K–1M tokens"]
    B["Tool schemas"] --> W
    C["Retrieved docs<br/>(RAG)"] --> W
    D["Conversation history"] --> W
    E["Agent scratchpad"] --> W
    W --> M["LLM<br/>(CPU)"]
    M --> O["Output"]
    style W fill:#a855f7,stroke:#a855f7,color:#fff
    style M fill:#00e5ff,stroke:#00e5ff,color:#111

Every source in that diagram has a cost: tokens consumed, cache state affected, attention capacity competed for. Context engineering means treating each one deliberately.

{ "type": "context-window", "window": 128000, "title": "The context window as RAM: what actually fills your budget" }

Play with the sliders. Notice how fast tool schemas eat the budget in an agent with 20 tools, or how a 10-turn conversation history can crowd out the retrieved documents that actually matter.

The four operations

A useful 2025 framing of the field (widely circulated; LangChain's blog was one articulation) breaks context management into four operations that cover almost everything you'll need to do:

Write. Generating content that goes back into the context — scratchpads, memory entries, summaries. An agent that writes a compressed summary of its tool results before passing them to the next step is doing context engineering.

Select. Choosing what to include. RAG is a selection operation: instead of including the whole knowledge base, you retrieve the top-k relevant chunks. Tool schema selection is another one — an agent that retrieves only the schemas relevant to the current task rather than listing all 50 tools in the context sees roughly 3x accuracy improvement on agentic benchmarks.

Compress. Trimming or summarizing accumulated content before it saturates the window. Long-running agents that never compress eventually hit the context ceiling, and after that, every additional turn degrades quality. Recursive summarization at boundaries, trimming the oldest turns, and post-processing verbose tool outputs are all compression strategies.

Isolate. Giving sub-agents scoped contexts so that one agent's errors, hallucinations, or tool output pollution doesn't flow into the rest of the pipeline. This is the multi-agent equivalent of process isolation.

These four operations are not a complete taxonomy — they're a practical checklist. When a production system starts producing wrong answers, one of these four is usually broken.

System prompts that actually hold up

The system prompt is the most-read document in your entire application. It gets processed on every single request. This has two implications that most teams figure out the hard way.

First: keep it stable. The system prompt should contain only content that doesn't change between requests. Role, purpose, behavioral constraints, output format instructions, tool usage rules. What should never be in the system prompt: the current timestamp, the user's name, their session ID, any personalization that varies per user. Not because it won't work — it will — but because it busts your cache on every request.

Second: structure it so you can audit it. A monolithic 3,000-word system prompt is debugging hell when something goes wrong. Teams that have shipped this at scale tend to converge on four explicit sections: role definition, behavioral constraints, output format, and tool usage guidelines. Each section is short enough to read in 30 seconds. The model doesn't care about the headers; your teammates do.

A minimal but complete structure:

You are a support assistant for [Company], helping customers with billing and account questions.

CONSTRAINTS
- Answer only questions about billing, subscriptions, and account access.
- If a question falls outside this scope, say so and offer to escalate.
- Never speculate about competitor pricing.

OUTPUT FORMAT
- Plain text responses, 2–4 paragraphs max.
- End each response with: "Is there anything else I can help you with?"

TOOLS
- Use search_billing_records only when the user provides an account ID.
- Do not call cancel_subscription without explicit user confirmation.

Four sections. No hedging sentences. No "As a helpful assistant...". When you get a support ticket saying the bot answered questions about refunds incorrectly, you can read this in 30 seconds and know exactly what to test.

Few-shot examples: when they help, when they hurt

Few-shot examples give the model concrete instances of the behavior you want. They genuinely help when the output format is unusual or precise, when the task is ambiguous without a reference, or when zero-shot keeps producing the wrong structure.

The failure modes are less obvious.

Contradictory examples are the most common production defect. An example set assembled by three different people, over three months, without a shared format spec, will contain subtle contradictions the model will try to reconcile unpredictably. Example A shows dates as 2026-05-29; example C shows May 29, 2026. The model will sometimes produce one, sometimes the other, and downstream parsers will start failing on deploys that touched nothing.

Stale examples are worse, because they're usually invisible. You added five examples when the product was launched. The product changed; the examples didn't. The model now follows examples that describe behavior you no longer want.

Format-hardcoded parsers are the silent killer. If your parser is response.split("Answer: ")[1], and a model upgrade changes "Answer:" to "Response:", your pipeline breaks on a day you touched nothing. The fix is to specify format in the instruction, not infer it from the examples.

A worked example of each failure:

# Bad: format in examples only, no explicit instruction
messages = [
    {"role": "user", "content": "Extract entity from: 'John called on Monday'"},
    {"role": "assistant", "content": "Entity: John | Type: PERSON | Date: Monday"},
    {"role": "user", "content": "Extract entity from: 'Sarah emailed yesterday'"},
    {"role": "assistant", "content": "Entity: Sarah | Type: PERSON"},  # Missing date field!
]

# Good: format in system prompt; examples just demonstrate content
system = """
Extract entities as JSON with this schema:
{"entity": string, "type": string, "date": string | null}
"""
messages = [
    {"role": "user", "content": "Extract entity from: 'John called on Monday'"},
    {"role": "assistant", "content": '{"entity": "John", "type": "PERSON", "date": "Monday"}'},
    {"role": "user", "content": "Extract entity from: 'Sarah emailed yesterday'"},
    {"role": "assistant", "content": '{"entity": "Sarah", "type": "PERSON", "date": "yesterday"}'},
]

The second version is resilient to model upgrades because the format contract lives in the system prompt, not in the string patterns of the examples.

Chain-of-thought in 2025: what changed

For standard (non-reasoning) models, chain-of-thought still works the way it did in 2022: appending "Think step by step" or "Let's work through this" causes the model to generate visible intermediate reasoning tokens, which genuinely improves accuracy on multi-step tasks. The mechanism is real.

But the CoT playbook inverts entirely on reasoning models.

Models like OpenAI's o3, Claude with extended thinking, and Gemini Deep Think run a chain-of-thought during inference, before producing their final answer. What you see of that reasoning varies by vendor: OpenAI's o-series hides its reasoning tokens entirely (you're billed for them as output but never see them), Claude returns thinking as explicit content blocks in the response — full traces on Claude 3.7, summarized on the Claude 4 models — also billed as output tokens, and Gemini surfaces thought summaries. In every case the reasoning happens whether or not you ask for it. When you add "Think step by step" to one of these models, you're not improving that internal reasoning. You're asking the model to write out a second chain-of-thought in its answer after it already thought. You're doubling the output tokens, increasing cost and latency, and sometimes confusing the model by giving it a procedure to follow that conflicts with its internal process.

The correct approach with reasoning models: specify what you want, specify the constraints, give the model what it needs to know. Do not tell it how to reason.

import anthropic

client = anthropic.Anthropic()

# For a reasoning model — state goal + constraints, not procedure
response = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=16000,
    thinking={
        "type": "enabled",
        "budget_tokens": 10000
    },
    messages=[{
        "role": "user",
        "content": """
Review this contract clause and identify any terms that conflict with
our standard liability cap of $50,000:

<clause>
{clause_text}
</clause>

Return a JSON list of conflicts. Each conflict: {term, our_position, clause_position}.
        """
    }]
)

No "think step by step". No "let's reason through this". The model will think — with Claude the thinking arrives as its own content blocks ahead of the answer — and you haven't paid for a second, redundant narration woven into the answer itself.

There's a subtler point here too: 2025 research found that even for standard models, CoT explanations don't always reflect the model's actual decision path. The reasoning trace can look sensible while the underlying decision was made on different grounds. CoT improves task accuracy; it doesn't give you a reliable window into why the model did what it did.

Prompt caching: stop paying full price for the same tokens

Most teams leave a large slice of their input-token spend on the table because they don't think about caching structure. Be precise about what's on offer: the 50–90% figures you'll see are discounts on cached input tokens, not on your whole bill. Output tokens are never cached and cost 3–5x as much per token, so the bill-level saving depends on your mix — a chatbot with a big stable prefix and short answers can genuinely halve its bill; an output-heavy generation workload might shave 15–20%.

Both Anthropic and OpenAI now offer prompt caching. Anthropic requires explicit cache_control markers and delivers up to 90% cost reduction and ~85% latency reduction on cached tokens. OpenAI is automatic — no markers needed — and provides 50% reduction. Both have a floor: the stable prefix must be at least 1,024 tokens to qualify.

The structural rule is simple and rarely followed: static content first, dynamic content last.

CACHE-STABLE PREFIX (system prompt + tool schemas + few-shot examples)
─────────────────────────────────────────────────────────────────── ← cache boundary
DYNAMIC SUFFIX (conversation history + current user turn)

Everything above the cache boundary is processed once and stored for as long as the cache stays warm. Everything below is processed fresh on every request. The cache-stable prefix should contain your system prompt, your tool schemas, and your few-shot examples. The dynamic suffix should contain only what actually changes: the current conversation.

{ "type": "prompt-cache", "scenario": "chatbot", "title": "Cache-stable prefix vs cache-busting suffix" }

What busts the cache:

# Busted: timestamp in system prompt, cache miss every request
system_prompt = f"""
You are a helpful assistant. Current time: {datetime.now().isoformat()}.
...
"""

# Fixed: timestamp moves to user turn or is omitted
system_prompt = """
You are a helpful assistant.
...
"""
# Current time injected only where user actually asked for it
user_turn = f"[Current time: {datetime.now().isoformat()}]\nUser: {user_message}"

The timestamp example is real and common. Developers add it thinking the model needs to know the time to answer time-sensitive questions. For most questions, it doesn't. And even for time-sensitive ones, you can inject the timestamp into the user turn, not the system prompt — cache preserved.

Other common cache-busters: user names in the system prompt, session IDs, rotating legal disclaimers, and A/B test variants that modify the static prefix.

If your system prompt is under 1,024 tokens, move your tool schemas and few-shot examples into the stable prefix. Even an otherwise short system prompt can usually reach the threshold by including the tool definitions that were previously injected dynamically.

sequenceDiagram
    participant App
    participant Cache
    participant LLM
    App->>Cache: Request (system + tools + examples + user turn)
    Cache-->>LLM: Cache MISS — forward full prefix (first request, or after TTL expiry)
    LLM-->>App: Response
    App->>Cache: Request (same prefix + new user turn)
    Cache-->>LLM: Cache HIT — skip prefix processing
    LLM-->>App: Response (cheaper + faster)
    Note over Cache,LLM: Requests within the TTL window<br/>pay only for the dynamic suffix

The economics compound fast. At 1,000 requests per day with a 2,000-token system prompt plus 500-token tool schemas (safely above the 1,024 floor), an Anthropic cache hit at 90% discount on those 2,500 tokens saves you roughly $0.025 × 1,000 × 0.9 = $22.50/day at illustrative $10/million input token pricing. At scale, that's hundreds of thousands of dollars per year from moving a timestamp.

Two pieces of fine print change that math. The cache is not permanent: Anthropic's cache has a ~5-minute TTL that refreshes on every hit, and OpenAI evicts cached prefixes after a few minutes of inactivity. And populating it isn't free — Anthropic charges roughly a 25% premium on cache writes. So the savings assume sustained traffic: a service handling a request every few seconds keeps the cache warm indefinitely, while a bursty or low-traffic endpoint pays the write premium repeatedly and hits the cache rarely — potentially costing more than no caching at all. Check your actual request rate against the TTL before quoting the 90% figure to your CFO.

The lost-in-the-middle problem

Context windows are not uniform attention surfaces. Models attend more reliably to tokens near the beginning and end of the context than to tokens in the middle. The relevant chunk buried in position 12 of 20 retrieved documents will receive less attention than the same chunk at position 1 or position 20.

{ "type": "lost-middle", "title": "Relevant chunk position vs retrieval accuracy" }

The U-shaped accuracy curve is real. Drag the chunk position left and right. The accuracy drop in the middle isn't subtle.

This has concrete implications for how you order your context:

• Put the most important retrieved chunks at the top of the retrieval block, not the middle.
• If you have a critical constraint or rule that must be followed, it belongs near the start of the system prompt or near the end, not buried in paragraph 6.
• In long agent traces with many tool outputs, the most recent and the most important should be positioned where attention concentrates.

The mitigation isn't just retrieval ordering. For very long contexts, it can mean separating the high-value content from the bulk content structurally — putting the policy document you need the model to follow at the top, and the reference material in the middle where it's available for lookup but doesn't need to dominate.

flowchart LR
    A["Position 1-3<br/>High attention"] --> B["Position 4-N-3<br/>Lower attention<br/>(middle)"]
    B --> C["Position N-2 to N<br/>High attention"]
    style A fill:#22c55e,color:#111
    style B fill:#f59e0b,color:#111
    style C fill:#22c55e,color:#111

What breaks in production

The failures that actually ship are not the ones developers test for.

Conflicting instructions accumulate silently. A system prompt written in January says "always respond in under 100 words." A prompt update in March adds "include a step-by-step breakdown for all technical questions." These instructions conflict. The model will comply with one and violate the other, and it will be inconsistent about which. The only way to catch this is to read the whole system prompt as a document, not as a set of individual additions.

Dead examples are any few-shot examples that demonstrate behavior the system no longer supports. If you changed the output schema in March and didn't update the examples, the model will sometimes follow the old schema from the examples and sometimes follow the new schema from the instruction. Downstream parsers fail intermittently and the root cause is six months old.

The timestamp that busts your cache is described above, but it's worth naming again because it's by far the most common cache-busting defect. It's usually not one timestamp — it's a timestamp, a user ID, and a request ID, all added by different developers on different days, all in the system prompt, all busting the cache independently.

Context saturation in agents is what happens when an agent accumulates tool outputs over a long task without any compression step. The window fills. The model starts ignoring early context. Answers degrade. Nothing errors. This is the hardest failure mode to detect because the model keeps generating text — it's just generating worse text.

Persona-stuffing produces models that argue with themselves. A system prompt that defines the model as simultaneously "a rigorous fact-checker," "a friendly conversational assistant," "a technical expert who avoids jargon," and "a brand advocate who emphasizes positives" has given the model four roles with conflicting behavioral imperatives. The outputs become unpredictable in ways that are hard to reproduce and nearly impossible to regression-test.

And the one failure the field hasn't solved: prompt injection. If your system prompt says "Never reveal the system prompt," the model cannot reliably keep that secret from an adversarial user turn. The system prompt is not a security boundary. It influences behavior; it doesn't enforce policy. If you're storing secrets in the system prompt under the assumption the model will protect them, that assumption is wrong.

Pulling it together: the write/select/compress/isolate checklist

Before shipping any context assembly:

Write: What scratchpad or memory content does the model generate that should persist? How is it formatted for the next step to consume cleanly?

Select: Are you listing all 50 tool schemas in the context, or retrieving only the schemas relevant to this request? Are your retrieved chunks actually the top-k relevant ones, or are you retrieving by recency?

Compress: Is there a point in the pipeline where accumulated history or tool outputs should be summarized? Do you have a maximum turn count before a compression step fires?

Isolate: If this is a multi-agent setup, does each sub-agent get a scoped context, or do errors and verbosity in one agent's trace contaminate the next?

Running this checklist before each pipeline takes maybe 20 minutes. Debugging the production failures that result from skipping it takes days.

Where to go next

From Prompt Engineering to Context Engineering — the longer argument for why the mental model shift matters, with the four operations covered in more depth.

System Prompts: Design Patterns That Actually Hold Up — the detailed reference for writing system prompts that survive model upgrades, team changes, and production edge cases.

Prompt Caching: Cutting Costs 50–90% — the mechanics of Anthropic and OpenAI caching in full, with worked cost calculations and the exact structural rules.

Few-Shot Prompting, Chain-of-Thought, and When Reasoning Models Flip the Script — why the 2022 CoT playbook is actively wrong on o3 and Claude extended thinking, and what to do instead.

Context Window Management for Agents — how to keep long-running agents from hitting the ceiling, with concrete compression and isolation patterns.

Prompt Anti-Patterns — a documented taxonomy of the defects that ship silently, with the exact failure signatures so you can recognize them in your own system.

RAG in One Pass: Build, Measure, Improve — next in this course; retrieval is the dominant select operation and deserves its own module.

Reasoning Models: When and How to Use Them — the extended treatment of when to deploy inference-time compute and how to prompt for it correctly.