Context Windows, KV Cache, and the Reality Behind 1M Token Claims

A team ships a document Q&A system. Their flagship feature: "ask anything about our 400-page technical manual." They dump the entire manual into a single API call — 180K tokens — because the model advertises a 200K context window. The demo sails through review. Two months in, a customer escalation lands: the bot quoted the warranty terms from page 212 and got them backwards, and the spot-check that follows finds every answer sourced more than 60 pages from either end of the manual is a coin flip. No request ever errored. The context fits. The model just stopped reliably attending to the middle.

That is the gap this article closes: the difference between what a context window number means on a marketing page and what it means on a GPU.

What the KV cache actually is

Every transformer layer computes attention by comparing a query (Q) against keys (K) and then blending the corresponding values (V). For the first token in a new generation, there is nothing to compare against, so the model does a full forward pass over the entire input — this is the prefill phase. It is embarrassingly parallel; modern hardware can process thousands of input tokens simultaneously.

After prefill, generation is autoregressive: one new token at a time. To compute attention at step N, the model needs the K and V tensors for every prior token. Without caching, that means re-running the full forward pass over all N tokens to get their K and V representations again — O(N) compute per new token, making generation cost grow quadratically with sequence length.

The KV cache short-circuits this. After prefill, the serving engine retains the K and V tensors from every layer in GPU memory. Each decode step runs the forward pass for just the one new token, appends its K/V, and attends against the cache. What the cache eliminates is the O(N) recomputation of every prior token's K/V projections and MLP passes. What remains per step is one token's forward pass plus an O(N) read of the cached keys and values — the whole cache streams through the GPU on every step, which is exactly why decode ends up memory-bandwidth-bound. Fast, at the cost of memory.

This trade-off is the central tension of long-context serving. The KV cache does not compress or summarize; it stores every K and V tensor verbatim. And those tensors are large.

The memory math

The size of the KV cache for a single sequence is:

KV cache bytes = 2 × L × H_kv × d_head × T × element_size

Where L is the number of layers, H_kv is the number of KV heads, d_head is the head dimension, T is the number of tokens, and element_size is 2 bytes for fp16 or 1 byte for fp8.

For Llama 3 70B with GQA (80 layers, 8 KV heads, head_dim 128):

One token: 2 × 80 × 8 × 128 × 2 = 327,680 bytes ≈ 0.31 MB
128K tokens: 128,000 × 0.31 MB ≈ 39.3 GB

The model weights in fp16 are ~140 GB. On a single H100-80GB GPU, you cannot even load the weights, let alone the cache. Even with fp8 weights (~70 GB) and fp8 KV cache (~20 GB for 128K context), you are at 90 GB — over the limit. A single H100 does not serve this model at 128K at all; you need 2-way tensor parallelism, KV offloading, or cache eviction. Drop to 32K and the fp8 cache is ~5 GB per sequence, so the ~10 GB left after weights fits a batch of 2. Serving a batch of 8 at 128K means 8 × 20 GB = 160 GB of cache on top of 70 GB of weights — 230 GB total, a 3–4 GPU tensor-parallel setup.

This is why long-context batch sizes collapse. The GPU is not doing more compute per request at 128K — it is hemorrhaging memory. And memory-bounded workloads on GPUs are slower and more expensive than compute-bounded ones.

{
  "type": "kv-cache",
  "model": "70b",
  "context": 128000,
  "batch": 1,
  "gpu": "h100-80",
  "title": "KV cache vs model weights at 128K context"
}

Why GQA exists

Standard multi-head attention (MHA) allocates independent K and V projection matrices for each of H query heads. For H=64 heads, you store 64 separate K vectors and 64 separate V vectors per token per layer. The query heads need this: each head can specialize in different patterns (syntactic dependencies, coreference, positional proximity) because their Q projections differ. But the argument for needing equally diverse K and V representations is weaker.

Grouped-Query Attention (GQA) shares a single K/V projection across a group of G query heads. H query heads divided into groups of G means H/G KV heads total. Llama 3 70B uses 64 query heads and 8 KV heads — a reduction factor of 8. Since KV cache size scales with KV head count, this cuts the cache from the MHA baseline by 8×.

flowchart TD
    subgraph MHA["Multi-Head Attention (H=8: caches 8 K/V pairs)"]
        Q1[Q₁] --> A1[Attn]
        K1[K₁] --> A1
        V1[V₁] --> A1
        Q2[Q₂] --> A2[Attn]
        K2[K₂] --> A2
        V2[V₂] --> A2
        Q3[Q₃] --> A3[Attn]
        K3[K₃] --> A3
        V3[V₃] --> A3
        Q4[Q₄] --> A4[Attn]
        K4[K₄] --> A4
        V4[V₄] --> A4
        Q5[Q₅] --> A5[Attn]
        K5[K₅] --> A5
        V5[V₅] --> A5
        Q6[Q₆] --> A6[Attn]
        K6[K₆] --> A6
        V6[V₆] --> A6
        Q7[Q₇] --> A7[Attn]
        K7[K₇] --> A7
        V7[V₇] --> A7
        Q8[Q₈] --> A8[Attn]
        K8[K₈] --> A8
        V8[V₈] --> A8
    end

    subgraph GQA["Grouped-Query Attention (8Q, 2KV: caches 4× less)"]
        GQ1[Q₁] --> GA1[Attn]
        GQ2[Q₂] --> GA1
        GQ3[Q₃] --> GA1
        GQ4[Q₄] --> GA1
        GK1["K₁ (shared)"] --> GA1
        GV1["V₁ (shared)"] --> GA1
        GQ5[Q₅] --> GA2[Attn]
        GQ6[Q₆] --> GA2
        GQ7[Q₇] --> GA2
        GQ8[Q₈] --> GA2
        GK2["K₂ (shared)"] --> GA2
        GV2["V₂ (shared)"] --> GA2
    end

    style GK1 fill:#a855f7,color:#fff
    style GV1 fill:#a855f7,color:#fff
    style GK2 fill:#a855f7,color:#fff
    style GV2 fill:#a855f7,color:#fff

Quality loss from GQA is minimal in practice — evaluations on standard benchmarks show sub-1% degradation versus MHA. GQA is now the default in Llama 3, Qwen 2.5, Gemma 2, and Mistral. If you are running a model released after mid-2024, it almost certainly uses GQA or something more aggressive. The detail matters for memory planning: when vendors publish KV cache specs, check whether they are reporting MHA or GQA memory footprint — the difference is 4–8×.

There is an extreme version called Multi-Query Attention (MQA), which uses a single K/V pair shared across all heads. It slashes cache size by H× but at more noticeable quality cost. MQA is mostly seen in older or smaller models.

Multi-head Latent Attention (MLA) takes a different route than head-sharing: it compresses keys and values into a single low-rank latent vector per token, caches only that latent, and reconstructs per-head K and V on the fly during attention. This is the signature architecture of DeepSeek-V2 and V3 — not GQA — and it cuts cached bytes per token several-fold below what GQA achieves (on the order of 0.07 MB per token for DeepSeek-V3, versus the ~0.31 MB worked above for Llama 3 70B), at the cost of extra decompression compute per decode step. If your memory budget is dominated by the KV cache, MLA is the most aggressive production-proven reduction shipping in a frontier-class open model as of mid-2026.

How context length gets set — and why "1M tokens" is not a retrieval guarantee

A model's context window is determined by three interacting factors: the positional encoding scheme, the sequence lengths seen during training, and the fine-tuning data used to extend context post-training.

Modern models use Rotary Positional Embeddings (RoPE), which encode position as a rotation in 2D subspaces of the embedding space. RoPE has a nice property: it naturally represents relative distances between tokens rather than absolute positions, which makes it more generalizable than the learned absolute embeddings used in GPT-2. The deeper discussion on the mechanics lives in Embeddings and Positional Encoding.

What matters here: RoPE has a frequency spectrum tied to the positions seen during training. If a model was pre-trained on sequences up to 8K tokens, attending to position 100K requires a rotation frequency that was never optimized during training. The fix — RoPE scaling techniques like YaRN and LongRoPE2 — adjusts the frequency spectrum to extend the effective window. But extension comes with a quality tax. A 2× extension is generally safe with careful fine-tuning; 8× requires significant effort to recover retrieval quality at the far end.

This is why marketing numbers and real performance diverge. A vendor who fine-tunes a base model on 1M-token documents and reports pass@1 on a handful of needle-in-haystack probes can legitimately say "1M context." What they do not say: fine-tuning data is sparse at extreme lengths, so the model has seen very few examples of reasoning that requires attending to position 900K.

flowchart LR
    TL["Training length\n(e.g. 8K tokens)"] --> RPS["RoPE scaling\n(YaRN, LongRoPE2)"]
    RPS --> FT["Long-context\nfine-tuning\n(sparse examples)"]
    FT --> MW["Marketed window\n(e.g. 1M tokens)"]
    MW --> ER["Effective recall\n(degrades past ~32–64K\nin practice)"]

    style MW fill:#ffaa00,color:#0a0a0f
    style ER fill:#ff2e88,color:#fff

Benchmarks from mid-2026 on models marketed at 128K context consistently show reliable retrieval to roughly 32K–64K, with degradation accelerating past that point. The 1M-token claims from Gemini 1.5 Pro and Claude 3.x hold for structured retrieval tasks (finding a named entity, extracting a date) but fall apart for subtle cross-context reasoning that depends on synthesizing information spread across the full window.

The lost-in-the-middle problem

Even within the range where a model reliably retrieves single facts, there is a positional bias that matters for system design. The phenomenon is called lost in the middle: given a long context with the relevant passage in the middle, recall is consistently worse than when the same passage appears at the beginning or end.

The mechanistic explanation connects to attention patterns. Early layers attend strongly to the beginning of the sequence — the system prompt, the initial user turn — creating a primacy effect. Deep layers develop a recency bias, attending disproportionately to the most recent tokens. Tokens buried in the middle receive less attention mass on average.

There is also the attention sink problem: specific tokens — the BOS token, prominent punctuation — accumulate disproportionate attention weight regardless of their semantic relevance, documented mechanistically in 2026 research. Every head that attends to an attention sink is a head that is not attending to your information.

The practical consequence: if you are placing a 200K-token context into a model, information in positions 50K–150K is at highest risk of poor recall. The mitigations are:

• Place the most critical instructions at the very top of the system prompt and repeat the key directive just before the final user message.
• Use context window management techniques to front-load the most relevant retrieved chunks.
• For any task where recall position matters, test recall at each position — not just average accuracy across positions.

{
  "type": "lost-middle",
  "title": "Retrieval accuracy vs. position in context"
}

Context window as RAM: filling the budget

Think of the context window as RAM for the model's working memory. Every byte costs you, and unlike RAM there is no swap — overflow is silent truncation or an error, depending on the provider. The budget has fixed allocations competing for the same space:

{
  "type": "context-window",
  "window": 128000,
  "title": "How 128K tokens get consumed in practice"
}

A typical production allocation for a 128K window:

• System prompt + tools — 2,000–8,000 tokens, depending on tool count and how verbose the system prompt is. Tool schemas in JSON are token-heavy; the trade-offs are covered in designing tool schemas.
• Conversation history — grows unboundedly until you implement compression or truncation. A chat session with 20 exchanges of 500 tokens each burns 10K tokens.
• Retrieved context (RAG chunks) — typically 2K–16K tokens of retrieved material, carefully ordered with the most relevant chunks first.
• Output reserve — you must leave headroom. An output running up against the context limit truncates mid-sentence.

When the window fills, something gets evicted. Some providers silently truncate the oldest messages; others throw an error. Neither outcome is good. The discipline of context engineering is about making that allocation intentional rather than emergent.

What breaks at long context

Throughput collapses. At 128K context, a single sequence occupies 40+ GB of KV cache on a 70B model at fp16, ~20 GB at fp8. On an 80 GB GPU with fp8 weights (70 GB) that leaves 10 GB — half of one fp8 sequence at full context. The request does not fit on the card; you shard across two GPUs, and even that pair serves one 128K request at a time. GPU utilization becomes memory-bandwidth-bound rather than compute-bound, which is where GPUs are least efficient. Throughput in tokens per second per GPU can drop 5–10× versus serving the same model at 4K context. If you are building a service where multiple users make 128K requests simultaneously, plan the GPU count accordingly — it is not a linear scale.

Prefill attention scales quadratically. The attention computation during prefill is O(N²) in sequence length — all tokens attend to all prior tokens simultaneously. Doubling input length quadruples the attention FLOPs (the linear layers, which dominate total compute until very long contexts, scale linearly). This is why several providers price long-context tiers at a higher per-token rate above roughly 200K — the quadratic term gets passed through — and why very long prompts see higher latency than simple token-count math predicts. The familiar input-vs-output price gap comes from the opposite side of the ledger: input tokens are cheap because prefill is parallel and compute-efficient, while output tokens cost more because decode is serial and memory-bound. Flash Attention (v2/v3) reduces the memory footprint of the attention computation by tiling it, but the arithmetic is still O(N²).

Recall is not uniform. As discussed above, the middle of a long context is the riskiest place to put critical information. But there is a subtler version of this: even end-of-context recall is imperfect when the sequence is extremely long. The most recent tokens do benefit from recency bias, but tokens in the last 5% of a 1M-token context are still 50K tokens from the end — well past the range where any current model has strong training signal.

Generation quality degrades at long context. A phenomenon separate from recall: the model's generation at position 100K in a 1M-token context has to carry forward a coherent thread through that entire history. Even if attention were perfect, the residual stream at that depth accumulates noise. Activation spikes in deep pre-norm transformers — partially addressed by peri-normalization in Gemma 2 and OLMo 2 — are more pronounced at long context.

KV cache quantization errors compound. Reducing KV cache from fp16 to fp8 halves memory usage, and for most tasks the quality loss is under 1%. But for tasks requiring precise recall of numerical values, code, or rare tokens, quantization errors in the K/V space can cause subtle degradation that is hard to detect without targeted evaluation.

KV cache compression: the emerging middle ground

Between "full fp16 KV cache" and "RAG instead of long context" there is a growing set of KV cache compression techniques that treat the cache as a first-class compressible object.

ChunkKV (NeurIPS 2025) partitions the KV cache into semantic chunks and evicts low-importance chunks based on attention scores from recent queries. Unlike token-level eviction (which breaks across semantic boundaries), chunk-level eviction preserves the coherence of retained content. Empirical results show <2% accuracy loss at 50% cache eviction on standard long-context benchmarks.

Attention Matching (2026) uses a small offline analysis of attention patterns to identify which token positions a given model consistently attends to, then builds a model-specific eviction policy. It is more accurate than heuristic methods but requires a profiling step per model.

These techniques are not yet universal defaults — vLLM, SGLang, and TensorRT-LLM have varying levels of support as of mid-2026 — but the trajectory is clear. Within a year, fp8 quantization plus adaptive eviction will likely be the default KV cache configuration for long-context production serving, substantially changing the memory math.

When to use long context vs RAG

The decision is not "long context or RAG" — it is "what retrieval strategy fits the structure of the information."

Scenario	Better choice	Why
Fixed reference document, holistic reasoning required	Long context	The answer depends on how multiple sections relate; retrieval fragments the reasoning
Large corpus, specific locatable facts	RAG	Needle-in-haystack recall is more reliable at 32K than at 500K; cost and latency are far lower
Frequently updated information	RAG	Re-indexing is cheap; re-prompting with a new 500K-token context is expensive
Conversation history with a known user	Compressed context or summarization	Episodic memory compresses well; retrieval from history works for most queries
Code over a large codebase	RAG with code-specific chunking	Codebases are too large to fit; file-level retrieval plus dependency tracing beats brute-force context stuffing
Legal documents requiring exact quote	Long context at moderate length	The exact language matters; chunking can split relevant clauses

The crossover point — where full-context recall becomes unreliable enough that RAG wins — is roughly 32K–64K for most frontier models in mid-2026. Beyond that range, the long context vs RAG decision article covers the quantitative framework in depth.

One number to internalize: at $15 per million input tokens (illustrative frontier pricing, mid-2026), a 500K-token context call costs $7.50 per request. If your system makes 10,000 such calls per day, that is $75,000 per day in input tokens alone — before any output costs. A RAG pipeline retrieving 4K tokens per query costs $0.06 per request at the same rate, or $600 per day for the same traffic. The retrieval index is not free, but the arithmetic is not close.

One correction any staff engineer will raise before you rip out the long-context design: prompt caching. Every major provider now discounts input tokens that repeat a previously seen prefix — roughly 90% off cached reads as of mid-2026 (illustrative; check current rates). If that 500K-token context is a fixed corpus sent on every call — the opening scenario's 180K-token manual is the canonical case — the per-request cost drops from $7.50 to roughly $0.75 after the first request, and the daily bill from $75,000 toward $7,500 plus cache-write overhead. Still more than 10× the RAG pipeline, but no longer absurd, and with none of RAG's chunking failure modes. The caveats: caching only pays when the prefix genuinely repeats (per-user contexts, frequently edited documents, and prompts whose variable part comes first get little or no hit), cache entries expire on provider-defined TTLs, and a cached read does nothing for recall — lost-in-the-middle applies to cheap tokens exactly as it does to expensive ones. The mechanics and break-even math live in prompt caching for cost optimization. Caching rescues long-context economics; it does not rescue long-context reliability.

Reading the vendor spec sheet honestly

When a model advertises a context length, these are the questions to ask before trusting it:

What is the training length vs the fine-tuned length? If the model was pre-trained at 8K and extended via RoPE scaling with fine-tuning at 128K, ask for the needle-in-haystack benchmark broken out by position — not just maximum context size.

Is the benchmark score averaged across positions? A model scoring 90% on a needle-in-haystack task might score 98% at position 0 and 70% at position 75K, averaging to 90%. Position-resolved recall curves are the right unit.

What is the effective throughput at maximum context? Vendors rarely publish batch size vs context length curves. For planning infrastructure, you need to know whether "128K context" means "128K at batch size 32" or "128K at batch size 1."

Does the advertised window include output tokens? Some providers count input + output toward the context limit; others count only input. A model with a 200K context window that generates 4K tokens per response effectively has a 196K input limit, but the spec sheet will say 200K.

The decision in practice

The practical framework for using context windows efficiently:

For inference serving, assume KV cache will be your binding memory constraint at long context, not model weights. Budget GPU memory as weights + (max_context × per_token_kv_size × batch_size). Use fp8 KV quantization unless you have evidence it degrades your specific task. Use GQA models for anything new — there is no good reason to deploy MHA at this point.

For application design, test needle-in-haystack recall at the specific context lengths and information positions your real use case requires — not at average positions over a synthetic benchmark. If your application places a long retrieved document in the middle of a long conversation history, benchmark exactly that layout. The general benchmarks will not tell you what you need to know.

And when the marketing says "1M tokens," translate that to: "will technically accept 1M tokens, may reliably attend to the first and last 50K, has uncertain behavior in between." That translation might save you from shipping the document Q&A system that sails through review and then gets the warranty terms on page 212 backwards — because the relevant clause is always in the middle of the manual.

For the deeper mechanical story on how attention actually computes those K/V interactions, see Self-Attention: The Mechanism That Reads Every Token Against Every Other. For the serving-side math — continuous batching, scheduling, and why static batching wastes GPU cycles — see Continuous Batching and Scheduling. And for the full infrastructure picture of how KV cache interacts with speculative decoding and serving engines, the KV cache deep dive for serving engineers has the detail you need.