Context window management for agents: select, compress, isolate

A production agent I worked on looked fine in testing. It ran 8 tool calls, answered a complex multi-document question, passed all evals. When we put it in front of real users and their sessions ran to 15-20 turns, the failure mode was subtle: the agent would answer correctly but then contradict a constraint it had acknowledged four messages ago, or re-fetch data it had already retrieved because it stopped "remembering" it had done so. The context window wasn't overflowing — we were at ~60% utilization. The problem was that no one was managing what was in that 60%.

This is the defining characteristic of context failures in production agents: they degrade silently. The model does not throw an error. It just becomes progressively less coherent, less consistent, and less accurate as the relevant signal gets diluted by accumulated noise. The agent-loop visualizer below shows the stall from the outside: an agent burning steps re-doing work it no longer remembers doing.

{ "type": "agent-loop", "scenario": "stuck-loop", "title": "An agent stuck re-doing work it forgot" }

The context accumulation problem

Think of the context window as a fixed-size whiteboard. At the start of a task it is almost empty. Each agent step writes on it: the reasoning trace, the tool call, the tool output, the observation. Nothing gets erased unless you explicitly erase it.

By step 10 of a coding agent, that whiteboard might contain:

• The original task specification
• Three rounds of code the model wrote and then revised
• Eight tool outputs from file reads, terminal runs, and test results
• The model's own reasoning steps narrated in prose

Most of this is historical. The model needs the current code state and the most recent error, not the full ancestry of every revision. But it is all there, occupying tokens, and the model attends to all of it on every generation step.

flowchart TD
    S1[Step 1: 2k tokens] --> S5[Step 5: 18k tokens]
    S5 --> S10[Step 10: 45k tokens]
    S10 --> S15[Step 15: 90k tokens]
    S15 --> WARN["70% of 128k window\n→ compression overdue"]
    S15 --> S20[Step 20: 112k tokens]
    S20 --> FAIL["87% of window\nquality degraded, approaching error"]

    style WARN fill:#ffaa00,color:#0a0a0f
    style FAIL fill:#ff2e88,color:#111

The growth is dominated by tool outputs. A single call to a code execution tool might return a full stack trace — 1,500 tokens. A web search tool returning full page content might return 8,000 tokens per call. Verbose tool outputs are the fastest-growing segment, and they also contain the most noise relative to signal. (The trajectory above assumes chunkier tool outputs than the back-of-envelope in the summary block, which lands step 15 at ~60k — a search-heavy agent pulling full page content hits the steeper curve easily.)

The context-window visualizer below turns this budget math interactive — drag the segments and watch which one tips the window over first.

{ "type": "context-window", "title": "Context budget by segment — drag to see overflow", "window": 128000 }

Select: what enters the context

The cheapest token is the one that never enters the window. Selection strategies keep irrelevant content out before it becomes a compression problem.

RAG-based tool selection

Most agent tutorials list every tool schema in the system prompt. At 5 tools this is fine. At 50 tools, you're injecting 10,000 tokens of schemas into every single agent step, most of which describe tools that are irrelevant to the current step in the current task.

RAG-based tool selection works exactly like document RAG: embed each tool's name and description, retrieve the top-k most similar to the current step's query, and inject only those schemas into the context. The model sees a small, focused set of tools and can reason about them clearly.

from openai import OpenAI
import numpy as np

client = OpenAI()

# Pre-embed all tool descriptions at startup
def embed(text: str) -> list[float]:
    resp = client.embeddings.create(model="text-embedding-3-small", input=text)
    return resp.data[0].embedding

TOOLS = [
    {"name": "read_file", "description": "Read file contents from disk", "schema": {...}},
    {"name": "run_tests", "description": "Execute the test suite and return results", "schema": {...}},
    {"name": "search_web", "description": "Search the web for current information", "schema": {...}},
    # ... 47 more tools
]

tool_embeddings = [(t, embed(f"{t['name']}: {t['description']}")) for t in TOOLS]

def select_tools(task_step: str, top_k: int = 5) -> list[dict]:
    """Retrieve the k most relevant tools for the current agent step."""
    query_emb = np.array(embed(task_step))
    scores = [
        (tool, float(np.dot(query_emb, np.array(emb))))
        for tool, emb in tool_embeddings
    ]
    scores.sort(key=lambda x: x[1], reverse=True)
    return [tool["schema"] for tool, _ in scores[:top_k]]

LangChain's 2025 research on this approach showed roughly 3× accuracy improvement over listing all tools when the tool library contained 20+ entries. The intuition is straightforward: when the model sees 50 tool schemas, it is simultaneously attending to 50 options on every generation step. When it sees 5, it can reason clearly about which one fits.

Semantic memory retrieval

An agent with access to persistent memory faces the same problem at the memory layer. Injecting the full memory store on every step is wasteful and noisy. Instead, retrieve only the memory entries relevant to the current step.

# Pseudocode — works the same way with any vector store (Qdrant, Pinecone, pgvector)
def get_relevant_memories(current_step: str, memory_store, top_k: int = 3) -> list[str]:
    results = memory_store.search(query=current_step, limit=top_k)
    return [r.content for r in results]

The agent memory architectures article covers the full taxonomy of episodic, semantic, and procedural memory — but from a context-management standpoint, the key principle is the same: retrieve what is relevant to this step, not everything you have ever stored.

Lost-in-the-middle awareness

Selection is also about placement. The lost-in-the-middle effect is well-documented: models attend more reliably to content at the start and end of long contexts. If you must include a long retrieved document, put the most critical section near the top of your RAG results, not buried in position 3 of 5. The lost-middle visualizer below makes this effect visceral — drag the "relevant fact" chunk to different positions and watch how accuracy changes.

{ "type": "lost-middle", "title": "Where you place the fact matters as much as whether it's there" }

Compress: shrink what is already in

Even with good selection, context grows. Compression strategies reduce what is already in the window.

Post-processing tool outputs

The highest-return compression target is tool outputs. A run_tests tool that returns 3,000 tokens of pytest output is almost always reducible to a 150-token structured summary:

def compress_tool_output(tool_name: str, raw_output: str, model: str = "gpt-4o-mini") -> str:
    """Post-process a tool output to a compact summary before adding to context."""
    compress_prompt = f"""You are compressing a tool output for an agent context.
Extract only: status (pass/fail/error), key results, any error messages or failure details.
Discard: verbose logs, passing test details, boilerplate output.
Output as structured bullet points, max 200 tokens.

Tool: {tool_name}
Output:
{raw_output[:4000]}"""  # cap input to prevent runaway costs

    resp = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": compress_prompt}],
        max_tokens=250,
    )
    return resp.choices[0].message.content

This adds a small LLM call, but using a cheap model (gpt-4o-mini at ~$0.15/1M input tokens, illustrative as of mid-2026) to compress a 3,000-token tool output that would otherwise be re-read by the main frontier model on every subsequent step is almost always worth it. The math: if the frontier model charges $15/1M input tokens and runs 10 more steps, that compressed output saves 2,800 tokens × 10 steps × $15/1M = $0.42 in downstream costs while adding 3,000 × $0.15/1M = $0.00045 in compression cost. The ratio is ~900:1.

Conversation history management

Conversation history is the other major growth segment. A sliding window approach keeps the most recent N turns verbatim and summarizes everything older:

def manage_history(
    messages: list[dict],
    max_verbatim_turns: int = 6,
    summary_model: str = "gpt-4o-mini"
) -> list[dict]:
    """Keep recent turns verbatim; summarize and compress older turns."""
    if len(messages) <= max_verbatim_turns * 2:  # *2 for user+assistant pairs
        return messages

    # Peel off the real system prompt — it must survive compression verbatim,
    # or the agent loses its role and rules after the first summarization pass.
    system_prompt = []
    if messages and messages[0]["role"] == "system":
        system_prompt = [messages[0]]
        messages = messages[1:]

    older = messages[:-max_verbatim_turns * 2]
    recent = messages[-max_verbatim_turns * 2:]

    # Summarize the older portion
    summary_text = client.chat.completions.create(
        model=summary_model,
        messages=[
            {"role": "user", "content": (
                "Summarize this conversation history for an agent. "
                "Preserve: decisions made, facts established, constraints set, current state. "
                "Discard: small talk, redundant attempts, verbose reasoning traces.\n\n"
                + "\n".join(f"{m['role']}: {m['content']}" for m in older)
            )}
        ],
        max_tokens=500,
    ).choices[0].message.content

    return [
        *system_prompt,
        {"role": "system", "content": f"[Conversation history summary]\n{summary_text}"},
        *recent
    ]

The compression trigger matters. Do not wait until the window is full. Set a threshold at 70-80% utilization and compress proactively. By 90% you are already in the degraded zone, and a context that just hit the limit will error out mid-turn.

sequenceDiagram
    participant ORCH as Orchestrator
    participant AGENT as Agent
    participant COMPRESS as Compressor

    loop Each agent step
        ORCH->>AGENT: current messages (check utilization)
        alt < 70% window
            AGENT->>AGENT: proceed normally
        else 70-80% window
            ORCH->>COMPRESS: summarize older history
            COMPRESS-->>ORCH: compressed summary
            ORCH->>AGENT: compressed messages
            Note over AGENT: ~40-50% window after compression
        else > 90% window
            Note over AGENT: already degraded — emergency truncation
        end
    end

Recursive summarization at checkpoints

For very long tasks, a checkpoint strategy works better than continuous rolling compression: at predefined milestones (every 10 steps, or after each major phase completes), summarize the entire accumulated context into a "state snapshot" and start a fresh context with that snapshot as the new system prompt prefix. This resets the context clock while preserving the essential state.

The state snapshot should answer four questions: What was the task? What has been done? What is the current state? What remains? Everything else can be discarded.

Build it or use the platform feature?

Before hand-rolling any of this, check what your stack already ships — as of mid-2026, most of it exists natively. Anthropic's context-editing and memory-tool betas auto-clear stale tool results from the context, which is the post-processing strategy above, productized. OpenAI's Responses API supports automatic truncation of older history. And agent frameworks bundle compaction: Claude Code auto-compacts as the window fills, and LangGraph ships summarization nodes you drop into the graph. For a standard agent loop, the platform feature is the right default — it is one flag instead of a compression pipeline you now own and debug.

Hand-rolled compression earns its keep in two situations: custom preservation rules (a generic auto-compactor does not know that your error messages must survive verbatim while passing-test logs can vanish) and per-segment accounting (platform compaction gives you a smaller context, not a breakdown of which segment was eating it). If neither applies, take the free version.

Isolate: sub-agents with bounded contexts

Selection and compression extend a single agent's effective horizon. Isolation solves a different problem: some tasks are inherently too long for any single context, and the compounding of selection and compression errors across 50+ steps introduces its own noise.

Multi-agent isolation means decomposing the task into subtasks, each handled by a sub-agent with a fresh, bounded context. The orchestrator passes a compact task brief to each sub-agent; the sub-agent returns a compact result summary. No individual agent ever accumulates the full context explosion.

flowchart TD
    ORCH[Orchestrator\n~5k token context]

    ORCH -->|"task: research X\n(500 tokens)"| R1[Research Agent\n~20k context]
    ORCH -->|"task: analyze Y\n(500 tokens)"| R2[Analysis Agent\n~20k context]
    ORCH -->|"task: write Z\n(500 tokens)"| R3[Writing Agent\n~20k context]

    R1 -->|"summary: findings\n(800 tokens)"| ORCH
    R2 -->|"summary: conclusions\n(600 tokens)"| ORCH
    R3 -->|"draft section\n(1200 tokens)"| ORCH

    ORCH --> FINAL[Final synthesis\n~8k total]

    style ORCH fill:#00e5ff,color:#0a0a0f
    style R1 fill:#0e7490,color:#fff
    style R2 fill:#0e7490,color:#fff
    style R3 fill:#0e7490,color:#fff

The orchestrator's context stays small because it only ever sees task briefs and summaries, never the full intermediate work. Each sub-agent's context is bounded to its specific subtask. The information passing between them is compact and deliberate — you choose what the next agent needs to know, rather than letting it inherit everything.

The multi-agent orchestration patterns article covers the broader architectural choices. From a context management standpoint, the key constraint on isolation is designing good inter-agent communication: what you pass between agents should be a semantic summary of the result, not a raw dump of the work product.

from anthropic import Anthropic

client_anthropic = Anthropic()

def run_sub_agent(task_brief: str, available_tools: list, max_steps: int = 10) -> str:
    """Run a sub-agent with a fresh, bounded context and return a compact result."""
    messages = [{"role": "user", "content": task_brief}]

    for step in range(max_steps):
        response = client_anthropic.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=2048,
            system="You are a focused sub-agent. Complete the task in the brief. "
                   "Return a compact, structured summary of your result.",
            tools=available_tools,
            messages=messages,
        )

        if response.stop_reason == "end_turn":
            # Extract the text result and return it
            return next(
                b.text for b in response.content if b.type == "text"
            )

        # Handle tool use (abbreviated)
        messages.append({"role": "assistant", "content": response.content})
        # ... process tool calls, append results ...

    return "[sub-agent: reached step limit without completion]"

Context scoping and schema hiding

A subtler form of isolation applies within a single agent: schema hiding — structuring your application state so the model only sees the fields relevant to its current task, not the entire state object.

If your agent has access to a large runtime state (user profile, order history, session data, preferences), injecting the full state object on every step wastes tokens and introduces noise. Instead, pass a view of the state scoped to the current step:

from pydantic import BaseModel

class FullApplicationState(BaseModel):
    user_id: str
    order_history: list[dict]  # potentially 200+ orders
    preferences: dict
    current_cart: dict
    payment_methods: list[dict]
    address_book: list[dict]
    # ...

def get_agent_view(state: FullApplicationState, task: str) -> dict:
    """Return only the state fields relevant to this task."""
    if "order" in task.lower():
        return {
            "recent_orders": state.order_history[-5:],  # last 5 only
            "current_cart": state.current_cart,
        }
    elif "payment" in task.lower():
        return {
            "payment_methods": state.payment_methods,
            "current_cart": state.current_cart,
        }
    # ... other task types
    return {"current_cart": state.current_cart}  # minimal default

This is a lightweight form of isolation that does not require spinning up a separate agent — it just controls what enters the context.

The numbers: when to use what

Back-of-the-envelope cost modeling for a 20-step coding agent, frontier model at $15/1M input tokens (illustrative, mid-2026):

Scenario A: No context management
  Average context at step 10: 60k tokens
  Average context at step 20: 110k tokens
  Total input tokens across 20 steps: ~1.1M tokens
  Cost (input only): ~$16.50 per task

Scenario B: Tool output compression + history sliding window
  Average context (compressed): 25k tokens throughout
  Total input tokens across 20 steps: ~500k tokens
  Cost of 20 compression calls (gpt-4o-mini): ~$0.09
  Net cost: ~$7.59 per task  →  54% reduction

Scenario C: RAG tool selection + compression + 4-sub-agent isolation
  Each sub-agent context: ~20k tokens, 5 steps
  Orchestrator context: ~8k tokens
  Total input tokens (all agents): ~420k tokens
  Cost of compression calls: ~$0.07
  Net cost: ~$6.37 per task  →  61% reduction, better quality

The quality gain from Scenario C is harder to quantify but real: each sub-agent operates on a clean, focused context rather than an increasingly cluttered one. Task completion rate in production tends to improve meaningfully with isolation, even when raw token counts are similar to a well-compressed single-agent approach.

One caveat on those numbers: all three scenarios price input at the flat uncached rate. The prompt caching article covers the other big cost lever, and it interacts with these strategies — not always in your favor. Prompt caches key on an exact stable prefix, and two of the techniques above work against that: per-step RAG tool selection changes the tools array every step (a guaranteed cache miss on the schemas and everything after them), and every history-summarization event rewrites the prefix, forcing one full-price re-read of what was previously a 0.1× cached read. Worked line: compressing a fully-cached 60k context to 25k saves 35k tokens per step at the cached rate ($0.05/step at $1.50/1M) but costs one uncached read of the new 25k prefix ($0.38) — payback in ~7 steps if you compress at checkpoints, never if you compress continuously. The practical rule: keep the system prompt and a small fixed fallback tool set in a stable cached prefix, place the per-step selected tools below it, and summarize at coarse checkpoints rather than every turn. Compression and caching stack — but only if you architect for both.

What breaks

Silent quality degradation before overflow

The worst failure mode is the one that looks fine. Context at 75% utilization on step 15 of a complex task produces outputs that are plausible but slightly less accurate, slightly less consistent with earlier constraints. Without careful eval comparison or step-level tracing, you will not notice until users complain — or until they stop using the product without complaining.

The mitigation is observability with per-segment token accounting. Log not just total context length but how many tokens came from each source at each step. When the "tool outputs" segment grows beyond ~30% of the window, you are accumulating history you should be compressing.

Tools like LangSmith, Langfuse, and Braintrust all capture the full messages array at each step — the key is structuring your logging to break it down by segment, which typically requires adding metadata to your messages before sending them.

Compression information loss

Summarization discards information. The model running the compression call makes judgment calls about what is "important" — and those calls can be wrong. A compression step that summarizes "the test suite passed" when the actual output was "7 of 8 tests passed, 1 flaked" introduces a subtle inaccuracy that can compound in later steps.

Mitigation: be specific in your compression prompts. Include explicit rules for what to preserve (error messages verbatim, status codes, counts, file paths) and what to discard (passing test logs, progress bars, boilerplate headers). Use structured output formats for compressed summaries so downstream parsing is reliable.

Sub-agent communication errors

With isolation comes handoff risk. If the summary a sub-agent returns is incomplete or subtly wrong, the orchestrator makes decisions based on that summary — and unlike a single-agent context where errors are visible, the orchestrator cannot verify what the sub-agent actually did.

Mitigation: structured result schemas instead of free-form summaries. Define what a sub-agent result looks like (status, key findings, artifacts produced, blockers) and use structured output to enforce it. The structured outputs article covers the mechanics of constraining sub-agent output reliably.

Tool selection miss

RAG-based tool selection can retrieve the wrong tools if the embedding model misunderstands the current step context. If the model needs a tool and it is not in the selected set, it will either fail the step or hallucinate a call to a tool that does not exist.

Mitigation: always include a small "fallback" set of general tools (e.g., "ask for clarification", "report inability to proceed") in the selected set regardless of relevance score. Also log tool selection decisions — if a task type consistently requires a tool that the retrieval step misses, you may need to improve the tool description or add explicit routing rules for that task type.

The decision in practice

Apply these three strategies roughly in order of implementation cost:

Start with compression — it requires no architecture changes, just post-processing tool outputs before adding them to messages and adding a history summarization step. The token and cost reduction is immediate.

Add RAG tool selection when your agent has more than ~10 tools. Below 10, the overhead is not worth it; above 20, it becomes material to both context size and answer quality.

Move to multi-agent isolation when tasks routinely exceed 15-20 steps or when you are hitting quality degradation despite good compression. This is the most effective strategy for long-horizon tasks but requires the most architectural work — clear task decomposition, inter-agent communication schemas, orchestrator logic.

The agent loop article covers how agents cycle through perceive-reason-act; this article is about managing the memory those cycles accumulate. The agentic RAG article covers the retrieval-side of selection in more depth if your bottleneck is the quality of what gets retrieved rather than the volume.

One thing production has taught repeatedly: the agents that hold up at scale are not the ones with the largest context windows. They are the ones whose designers thought carefully about what should be in the context at each step — and had the tooling to see when that thinking was wrong.