The KV Cache: GPU Memory Math Every Serving Engineer Must Know

Your product manager wants 128K context on the new chatbot feature. Your model already fills two A100s just for the weights. You run the math and realize a single 128K request needs 41 GB of KV cache on top of the 140 GB of model weights — and that's before you handle a second user. At 10 concurrent users, you're looking at 410 GB of KV cache for a model that maxes out at 160 GB of total HBM across two GPUs.

This is not a hypothetical. It is the conversation that happens the first time a team deploys a 70B model with long context enabled and hasn't calculated the KV cache budget in advance. OOM crashes at load. The fix requires understanding a formula.

What the KV cache actually is

During autoregressive decoding, the model generates one token per forward pass. Each forward pass runs attention: every token attends over every previous token. To compute attention for token n, the model needs the key and value matrices for tokens 0 through n-1. Without caching, that means recomputing those matrices from scratch every step — an O(n²) cost in both compute and time.

The KV cache is straightforward: after computing keys and values for a token, store them in GPU memory. On the next decode step, load the cached copies instead of recomputing. The compute cost drops from O(n²) to O(n) per step. The memory cost is fixed per token per layer, and it grows linearly as the sequence gets longer.

What is not cached: the query matrices. Queries only need to be computed for the current token position (the one being generated), so caching them saves nothing.

sequenceDiagram
    participant GPU as "GPU Compute"
    participant HBM as "HBM (KV Cache)"
    participant OUT as "Output"

    Note over GPU,HBM: Prefill (full prompt processed in parallel)
    GPU->>HBM: "write K,V for all prompt tokens"

    Note over GPU,HBM: Decode step 1
    HBM->>GPU: "load K,V for tokens 0..N"
    GPU->>GPU: "compute Q for token N+1, run attention"
    GPU->>HBM: "write K,V for token N+1"
    GPU->>OUT: "token N+1"

    Note over GPU,HBM: Decode step 2
    HBM->>GPU: "load K,V for tokens 0..N+1"
    GPU->>GPU: "compute Q for token N+2"
    GPU->>HBM: "write K,V for token N+2"
    GPU->>OUT: "token N+2"

The pattern is clear: the KV cache grows by one row per layer on every decode step, and the HBM bandwidth required to load it grows with sequence length. This is why decode is memory-bandwidth-bound — the dominant cost at each step is reading the accumulated KV cache from HBM, not the matmul itself.

The formula

KV bytes per token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element

The 2 is for keys and values separately. Everything else is a model architectural parameter you can read from the config. For BF16 (the standard serving dtype), bytes_per_element = 2. For FP8, it is 1. For FP16, also 2.

Let's work through the most common models engineers actually serve (as of mid-2026):

Model               Layers  KV-heads  head_dim  dtype   MB/token   Notes
────────────────────────────────────────────────────────────────────────────────
Llama 3.1 8B           32       8       128    BF16    0.131    GQA (8 KV vs 32 Q)
Llama 3.1 70B          80       8       128    BF16    0.320    GQA (8 KV vs 64 Q)
Llama 3.1 405B        126       8       128    BF16    0.504    GQA (8 KV vs 128 Q)
Mistral 7B             32       8       128    BF16    0.131    GQA
Qwen 2.5 72B           80       8       128    BF16    0.320    GQA
DeepSeek-V3            61       —       —     BF16   ~0.070    MLA (Multi-head Latent Attention — caches a compressed latent, formula below does not apply; see GQA section)
GPT-4o-class           —       —       —      —        —       proprietary; architecture undisclosed — budget as if 70B-dense per token

Formula for each model above:
  Llama 3.1 70B: 2 × 80 × 8 × 128 × 2 = 327,680 bytes = 0.320 MB/token
  Llama 3.1 8B:  2 × 32 × 8 × 128 × 2 = 131,072 bytes = 0.131 MB/token

Now scale to real workloads:

Scenario: Llama 3.1 70B at BF16, serving B concurrent requests at context C

  KV cache total = 0.320 MB/token × C tokens × B requests

  B=1,   C=4K:    1.3 GB     — fits easily
  B=10,  C=4K:    13 GB      — workable
  B=100, C=4K:    128 GB     — 1.6× a single H100 80 GB chip
  B=10,  C=32K:   102 GB     — tight
  B=10,  C=128K:  409 GB     — 5× a single H100; needs a multi-node cluster

  Compare: model weights = 70B params × 2 bytes = 140 GB (needs 2× H100 minimum)
  At B=10, C=128K: KV cache (409 GB) dwarfs the model weights (140 GB) by 3×.

This is the number that breaks teams. They calculate GPU requirements based on model weights, deploy, and discover at load that the KV cache doesn't fit.

Run the math yourself in the widget below. At BF16 the 70B weights alone (140 GB) blow through a single H100's 80 GB — flip the dtype to FP8 so the weights fit, then drag the context slider toward 32K and watch the KV bar cross the HBM line.

{ "type": "kv-cache", "model": "70b", "context": 4096, "batch": 8, "gpu": "h100-80", "title": "KV cache vs model weights on H100 80GB" }

Grouped-query attention: the architectural lever

Multi-head attention (MHA) creates a separate key-value pair per attention head. A model with 64 attention heads has 64 sets of KV matrices per layer — every one of which gets cached.

Grouped-query attention (GQA) assigns multiple query heads to share a single key-value head. Llama 3.1 70B has 64 query heads but only 8 KV heads — each KV head is shared by 8 query heads. The KV cache shrinks to 8/64 = 12.5% of the MHA equivalent. That 8x reduction is the only reason 128K-context serving is economically viable on current hardware.

Multi-query attention (MQA) is the extreme case: all query heads share one KV head. Shrinks the cache maximally but hurts quality on complex reasoning tasks, which is why it fell out of favor for large models.

Multi-head latent attention (MLA) — the DeepSeek-V2/V3 approach — attacks the problem from a different angle. Instead of reducing the number of KV heads, it projects keys and values into a single shared low-rank latent vector per token, and that latent is what gets cached; the per-head K and V are reconstructed from it at attention time. The formula changes shape: instead of 2 × kv_heads × head_dim elements per layer per token, you cache one latent of a few hundred elements. DeepSeek-V3 caches a 576-element latent per layer (512 compressed KV dims plus a 64-dim decoupled RoPE key), which at BF16 across its 61 layers works out to ~0.07 MB/token — under a quarter of Llama 3.1 70B's 0.32 MB/token, despite V3 being the larger model. That is why DeepSeek can serve 128K contexts with a fraction of the KV footprint of a GQA 70B-dense. The catch: like GQA, MLA is baked in at training time, and the low-rank reconstruction adds a small compute cost per attention call — a trade that pays off precisely because decode is memory-bandwidth-bound, not compute-bound.

flowchart TD
    subgraph MHA["MHA (e.g., GPT-2): Q-heads = KV-heads = 32"]
        Q1[Q₁] --> K1[K₁/V₁]
        Q2[Q₂] --> K2[K₂/V₂]
        QN[...32 heads...] --> KN[...32 KV pairs...]
    end

    subgraph GQA["GQA (e.g., Llama 3.1 70B): 64 Q-heads, 8 KV-heads"]
        QA[Q₁-Q₈] --> KA["K₁/V₁ (shared)"]
        QB[Q₉-Q₁₆] --> KB["K₂/V₂ (shared)"]
        QC[...8 groups...] --> KC["...8 KV pairs..."]
    end

    style MHA fill:#1e293b,color:#94a3b8
    style GQA fill:#1e293b,color:#94a3b8
    style K1 fill:#ff2e88,color:#fff
    style K2 fill:#ff2e88,color:#fff
    style KN fill:#ff2e88,color:#fff
    style KA fill:#0e7490,color:#fff
    style KB fill:#0e7490,color:#fff
    style KC fill:#0e7490,color:#fff

GQA is baked into the model architecture at training time. You cannot retrofit it onto an existing MHA checkpoint. When choosing between otherwise equivalent models, the one with more aggressive GQA wins on serving cost at long contexts.

PagedAttention: fixing the fragmentation problem

Even with GQA, the allocation strategy matters. The naive approach pre-allocates a contiguous block of GPU memory for each incoming request, sized at the maximum possible sequence length. A request with a 4K context on a system configured for 128K max gets 128K worth of KV cache memory reserved — 97% of which sits unused until the request finishes, at which point it is freed.

Under mixed-length traffic (the realistic case), this causes heavy fragmentation. At 100 concurrent requests, you might have 60% of your HBM reserved but mostly idle, while new requests queue because there is no contiguous free block large enough to satisfy the pre-allocation policy.

PagedAttention treats KV cache the same way an OS treats virtual memory: divide HBM into fixed-size pages (vLLM uses 16-token blocks by default) and allocate them on demand as the sequence grows. When a request finishes, its pages are freed immediately and returned to the pool. No fragmentation. No wasted reservation.

The practical impact: the original PagedAttention paper (Kwon et al., SOSP 2023) reported 2-4x higher throughput than FasterTransformer and Orca at equivalent hardware, primarily from eliminating fragmentation and enabling higher concurrent batch sizes. The PagedAttention mechanism is now the baseline that all serious serving engines implement — vLLM, SGLang, and TensorRT-LLM all use it or a variant of it.

KV cache quantization: the highest-leverage knob

The KV cache is a large blob of floating-point matrices sitting in HBM. You can quantize it independently of the model weights.

BF16 (default): 2 bytes per element. Full precision, maximum quality.

FP8: 1 byte per element. On H100/H200/B200 hardware, FP8 operations are hardware-native — the conversion is free at the hardware level. Quality loss on standard benchmarks (MMLU, HumanEval, GSM8K) is typically under 0.5% when applied to KV cache only. The memory savings are a clean 2x. As of 2026, FP8 KV cache quantization is the production default on H100/H200 deployments.

INT4: 0.5 bytes per element. Pushes capacity further but quality degradation becomes task-dependent. Fine for extraction and classification workloads; riskier for long multi-hop reasoning chains where small errors compound.

KV cache memory at different dtypes — Llama 3.1 70B, B=10, C=32K:

  BF16:  0.320 MB/token × 10 × 32,768 = 104.9 GB
  FP8:   0.160 MB/token × 10 × 32,768 =  52.4 GB   ← 2× capacity increase
  INT4:  0.080 MB/token × 10 × 32,768 =  26.2 GB   ← 4×, but quality risk

  On 2× H100 SXM (160 GB total HBM):
    BF16: model weights (140 GB) + KV cache (104.9 GB) = 244.9 GB  ← OOM
    FP8:  model weights (140 GB) + KV cache  (52.4 GB) = 192.4 GB  ← still tight
    Reality: FP8 weights + FP8 KV = ~70 GB weights + 52 GB KV = fits

In vLLM, you enable FP8 KV quantization with --kv-cache-dtype fp8. In SGLang, it is --kv-cache-dtype fp8_e5m2. The implementation handles quantization and dequantization at the attention layer boundaries; the rest of the model is unaffected.

Prefix caching: reusing what you already paid for

If two requests share an identical prompt prefix — the same system prompt, the same document being queried, the same tool definitions in an agent — computing and storing separate KV caches for both is wasteful. You paid once, you should serve both.

Prefix caching (called RadixAttention in SGLang; in vLLM opt-in via --enable-prefix-caching from v0.4 and on by default since the V1 engine in 2025) stores KV cache entries indexed by their token hash. When a new request arrives with a prefix that matches an entry in the cache, the matching pages are reused directly. The cache hit eliminates both the prefill compute cost and the KV memory for the shared portion.

For agent workloads where every request starts with a 4K-token system prompt containing tool definitions, prefix caching makes that 4K free after the first request. For a RAG pipeline where 10 parallel queries all include the same 8K retrieved document, only one copy lives in the KV cache. The effective memory savings depend directly on the shared-prefix ratio, which on well-structured agent workloads is often 70-90%.

The interaction with continuous batching is important: the scheduler must be aware of which pages are shared and pin them in the cache. vLLM and SGLang both handle this automatically. The risk is cache eviction under memory pressure — if the shared prefix gets evicted to make room for a new request, the next request that needs it triggers a recompute. Getting the cache eviction policy right for your workload (LRU vs LFU vs priority-weighted) matters at high concurrency.

What breaks

OOM crashes at load, not at idle. The common failure pattern: you test with one request and it works fine. Under 50 concurrent users, the KV cache fills HBM and the process crashes with a CUDA out-of-memory error. The fix is to calculate your memory budget before deploying, not after.

Safe concurrent requests at 4K context, Llama 3.1 70B, BF16 weights + BF16 KV:

  2× H100 80GB = 160 GB total HBM
  Model weights: 140 GB
  Available for KV: 160 - 140 = 20 GB
  KV per request at 4K: 0.32 MB × 4,096 = 1.31 GB
  Max safe concurrent requests: floor(20 / 1.31) ≈ 15 requests

  This is before vLLM's overhead (activation memory, CUDA context, ~2-3 GB)
  Real ceiling: ~10-12 concurrent requests
  
  Switch to FP8 weights (~70 GB) + FP8 KV (0.16 MB/token):
  Available for KV: 160 - 70 = 90 GB
  KV per request at 4K: 0.16 MB × 4,096 = 0.655 GB
  Max concurrent: floor(90 / 0.655) ≈ 137 requests

That 10x difference between BF16 weights + BF16 KV versus FP8 weights + FP8 KV is not hypothetical — it is the gap you see on H100 deployments.

Memory fragmentation without PagedAttention. If you are running an older serving stack (or wrote your own inference code without paged allocation), you will see effective utilization far below the theoretical maximum. Requests fragment the contiguous pool. The symptom is HBM reading as 80% utilized while new requests OOM — plenty of memory in aggregate, but no contiguous block.

Underestimating KV cache growth during the request. The KV cache for a request at prefill time is tiny — it only covers the input prompt. During generation, it grows with every output token. A request with a 2K prompt that generates 6K tokens ends up with 8K of KV cache. If your memory budget was calculated on prompt length only, long-output requests exhaust your headroom faster than expected.

Prefix cache eviction under pressure. Prefix caching saves memory on average but creates spikes when hot prefixes get evicted. A batch of 50 requests that all need the same system prompt gets a cache hit for request 1-49, then a cache eviction event that forces a recompute for requests 50+. If the recompute itself causes memory pressure, you can get a cascade. Monitor cache hit rate and eviction events in production.

Context length and token counting mismatch. Token counts depend on the tokenizer, not word counts. A 10,000-word document is roughly 13,000-15,000 tokens for most English-language models. If you calculate KV budgets in words, you underestimate by ~30-50%.

The serving engine view

vLLM's PagedAttention implementation is what made dynamic KV cache allocation the industry standard. The vLLM serving engine manages the KV cache as a pool of fixed-size blocks, handles eviction and swapping, and surfaces metrics (vllm:gpu_cache_usage_perc, vllm:num_preemptions_total) that tell you how close to the limit you are.

SGLang's RadixAttention extends PagedAttention with a radix tree (trie) over the token hash space, making prefix cache lookup O(prefix length) rather than O(total cache size). This matters for workloads with thousands of distinct but partially overlapping prefixes — agent systems with per-user system prompts, multi-tenant RAG with per-customer context. The RadixAttention paper reports substantial throughput gains — roughly 30% over contemporary vLLM on workloads with large shared prefixes, and far more on heavily branching agent traces (specifics drift with each release; the mechanism is what matters).

LMCache (2025) takes this further: a shared KV cache layer across multiple serving instances, enabling cross-engine reuse. If your multi-instance setup has one node that recently processed a popular document, another node can fetch its KV cache pages instead of recomputing. At enterprise scale with hundreds of serving replicas, this can meaningfully cut HBM pressure. LMCache integrates with both vLLM and SGLang.

For the relationship between KV cache and the decode phase's memory-bandwidth bottleneck, see prefill vs decode phases. For how continuous batching affects the rate at which new requests can enter the pool (and therefore the effective KV cache pressure over time), see continuous batching and scheduling.

The decision in practice

The decisions you actually make at deploy time, in the order they matter:

First: calculate your KV budget before you pick hardware. KV GB = 2 × layers × kv_heads × head_dim × bytes × context × batch_size / 1e9 Add model weight memory and leave 10-15% headroom for activation memory and CUDA overhead. If the number exceeds your HBM, either reduce context, reduce batch size, or switch to a model with more aggressive GQA.

Second: use FP8 KV cache on H100/H200/B200. The hardware supports it natively; the quality tradeoff is negligible; the 2x capacity increase is immediate. There is no reasonable argument for staying on BF16 KV on modern hardware if memory is any kind of concern.

Third: enable PagedAttention (vLLM) or equivalent. Every serious serving engine does this now. If you are writing custom inference code and not handling paged allocation, you are leaving 30-60% of your effective capacity on the floor due to fragmentation.

Fourth: turn on prefix caching if your workload has shared prefixes. In vLLM: opt-in via --enable-prefix-caching from v0.4, enabled by default since the V1 engine (2025). In SGLang: RadixAttention is on by default. Check your cache hit rate in the metrics. For agent workloads with large tool definition blocks or fixed system prompts, a 70%+ hit rate is realistic and translates directly to memory savings and lower TTFT.

Fifth: consider GQA when choosing a new model. If you are evaluating two models with similar quality and one uses more aggressive GQA, the KV cache advantage compounds at long contexts. A model with 8 KV heads serves the same traffic as a comparable MHA model with 1/8th the KV memory. At 32K+ contexts, this can be the difference between fitting on one node versus two.

The KV cache is not a detail you can defer to operations. It is a first-order constraint that determines what you can serve, at what context length, at what concurrency. Get the formula into your back pocket before the product manager asks for 128K.

For the full picture of what happens after the KV cache is warm — how requests are scheduled, how the GPU stays full, and what continuous batching actually changes — see continuous batching and scheduling. For the economics of translating memory efficiency into cost-per-token, see throughput vs latency: the GPU economics of LLM serving.