Prefill vs Decode: The Two Phases of Every LLM Request

Your production chatbot has a 600ms pause before it streams any text. You profile the server and find your request spent most of that window queued behind someone else's 30K-token prompt — which pinned the GPU at full compute utilization the entire time. Meanwhile, your p50 inter-token latency is fine. You're staring at the canonical prefill problem: head-of-line blocking. Adding GPUs buys capacity in proportion to spend, with no efficiency win — the pause just gets rarer. The actual fix is scheduling, not "buy more compute."

Understanding why requires understanding what the GPU is actually doing during each phase of an LLM request.

What happens during prefill

When a request arrives at a serving engine, the first job is to process the prompt. Suppose the prompt is 2,048 tokens. The model needs to compute key-value pairs for every one of those tokens across every attention layer, and produce the hidden state that will be used to predict the first output token.

The critical property is that this happens in parallel. All 2,048 tokens are processed simultaneously in one forward pass. The dominant operation is matrix multiplication: the 2,048-token sequence gets projected through the model's weight matrices in large, dense matmuls. On modern GPUs, large dense matmuls are exactly the operation for which the hardware is designed — the arithmetic intensity (FLOPS per byte of memory accessed) is high, and the GPU's tensor cores stay busy.

sequenceDiagram
    participant C as Client
    participant S as Serving Engine
    participant G as GPU

    C->>S: POST /v1/completions (prompt: 2048 tokens)
    S->>G: prefill pass: process tokens 1..2048 simultaneously
    Note over G: Large dense matmuls, high FLOP utilization
    G->>S: KV cache written for all 2048 tokens
    G->>S: first output token logits
    S->>C: first token (TTFT = time so far)
    loop decode: one token per iteration
        G->>S: load weights + KV cache from HBM
        G->>S: next token logits
        S->>C: stream token
    end

TTFT — time to first token — is entirely prefill time plus queueing delay. A 4K-token prompt might take 50–200ms to prefill on an A100; a 32K-token prompt can take 500–2000ms. Users experience this as the "thinking pause" before any text appears.

What happens during decode

After prefill, the model enters decode: it generates one token per step, feeds that token back as input, and repeats until it hits an EOS token or a maximum length. Each decode step is a forward pass with a sequence of length one — the single new token — plus the KV cache from all previous tokens.

Here's where the arithmetic flips. A single-token forward pass does almost no compute relative to the data it has to read. To generate one token, the GPU must:

1. Load all model weights from HBM — that's 16 GB for an 8B BF16 model, or 140 GB for a 70B model.
2. Load the KV cache for every previous token in the context — growing by one entry per step.
3. Run the forward pass — which, for a single token, is mathematically small.

The ratio of compute to memory bandwidth required is extremely low. This is the definition of memory-bandwidth-bound: the GPU has enormous math capacity sitting idle while it waits for data to arrive from HBM. An H100 SXM can do 989 TFLOPS of BF16 math. During single-request decode, you might use 5–20 TFLOPS of that — because the bottleneck is the 3.35 TB/s HBM bandwidth ceiling, not the math units.

Decode ceiling: 70B BF16, tensor-parallel across 2× H100 SXM
(140 GB of weights doesn't fit one 80 GB card — TP2 is the minimum)
  Weight load per step: ~140 GB total, ~70 GB per GPU
  Aggregate HBM bandwidth: 2 × 3.35 TB/s = 6.7 TB/s
  Max steps/sec: 6,700 / 140 ≈ 48 tok/s per request
  Real-world TP2 decode: ~30-40 tok/s at batch=1
  (kernel launch + all-reduce overhead eats the rest)

  FLOP utilization at 40 tok/s:
  ~140 GFLOPs / step × 40 steps/s = ~5.6 TFLOPS
  → 5.6 / 1,978 aggregate ≈ 0.3% of available FLOPS

This matters because most instincts about "my GPU is slow" lead to buying more FLOPS. But for decode, more FLOPS do not help. Only two things move the needle: higher HBM bandwidth (which is why the H100 → H200 → B200 progression matters for serving) or quantization (which reduces the bytes you load per step).

The two metrics you actually care about

TTFT and TPOT (time per output token, also called TBT — time between tokens) measure different user experiences and respond to different fixes.

Metric	Phase	User experience	What moves it
TTFT	Prefill	"Why is it thinking so long?"	Prompt length, chunked prefill, disaggregated prefill, GPU FLOPS
TPOT / TBT	Decode	"Why is the text streaming slowly?"	Batch size, HBM bandwidth, KV cache size, quantization
End-to-end latency	Both	Total time for complete response	Dominated by decode for long outputs

A common mistake: a team sees high end-to-end latency on their API, adds capacity, and wonders why it didn't help proportionally. Because the response was 2,000 tokens and the bottleneck was decode throughput, not prefill latency. Measuring only one number hides which phase needs work.

{ "type": "batching", "title": "How batching shifts the decode bottleneck", "mode": "continuous" }

The KV cache: prefill writes it, decode reads it

When the model processes the prompt during prefill, it computes key and value tensors for each attention head at each layer. These get written to a KV cache in GPU memory and reused on every subsequent decode step — rather than recomputing attention over the full history from scratch each time.

The KV cache is not free. Per-token KV memory scales as:

KV bytes per token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element

Llama 3.1 8B (GQA, 8 KV heads, 32 layers, head_dim=128, BF16):
  = 2 × 32 × 8 × 128 × 2 = 131,072 bytes ≈ 0.125 MB/token

At 100 concurrent requests × 4K context:
  = 100 × 4,000 × 0.125 MB = 50 GB KV cache
  + 16 GB model weights
  = 66 GB total → fits an H100 80GB, barely

Same workload with 32K context:
  = 100 × 32,000 × 0.125 MB = 400 GB → does not fit

This is why grouped-query attention (GQA) matters. MHA (multi-head attention) uses the same number of KV heads as query heads. GQA groups multiple query heads to share one KV head. Llama 3.1 70B uses 64 query heads but only 8 KV heads — an 8x reduction in KV cache memory, directly translating to either 8x more concurrent requests or 8x longer context for the same GPU.

The KV cache article has the full memory math and PagedAttention — which solves KV cache fragmentation. Understand that article before you size a production deployment.

{ "type": "kv-cache", "title": "KV cache VRAM vs model weights", "model": "70b", "context": 8192, "gpu": "h100-80" }

What limits throughput in each phase

The reason these phases need different optimizations becomes concrete when you look at what is actually saturated.

flowchart TD
    PF_BOUND["Prefill: compute-bound"]
    DC_BOUND["Decode: memory-bandwidth-bound"]

    PF_BOUND --> PF1["FLOPS determine speed"]
    PF_BOUND --> PF2["Bigger matmuls = better utilization"]
    PF_BOUND --> PF3["More concurrent prompts helps little"]
    PF_BOUND --> PF4["Fix: chunked prefill, faster GPU, flash attention"]

    DC_BOUND --> DC1["HBM bandwidth determines speed"]
    DC_BOUND --> DC2["Loading weights dominates time"]
    DC_BOUND --> DC3["Batching: amortize weight loads across requests"]
    DC_BOUND --> DC4["Fix: quantization, higher-bandwidth GPU, batching"]

    style PF_BOUND fill:#0e7490,color:#fff
    style DC_BOUND fill:#0e7490,color:#fff

Batching works for decode specifically because of memory bandwidth. If you serve 32 requests simultaneously during decode instead of 1, you still load the weights roughly once per step and run 32 token computations with that data. The memory load cost is amortized across 32 outputs. This is why continuous batching (covered in depth in the continuous batching article) achieves 3-24x throughput improvement over static batching — it keeps GPU batch slots filled.

Quantization improves decode more than prefill. Going from BF16 to FP8 halves the bytes loaded per decode step, roughly doubling decode throughput ceiling. The effect on prefill is smaller because prefill is already compute-bound — you're not waiting on memory loads. The quantization article covers GPTQ, AWQ, and FP8 in detail.

Chunked prefill: the first fix for TTFT at scale

Without any special handling, a 32K-token prefill monopolizes the GPU for potentially 500-1500ms. Every other request in the queue waits. This is head-of-line blocking, and it destroys p95 TTFT for all requests whenever a single long prompt arrives.

Chunked prefill (Sarathi-Serve, now default in vLLM) slices the prefill into fixed-size chunks — typically 512 or 1024 tokens — and interleaves them with decode steps from other in-flight requests. A 32K prompt becomes 32 chunks of 1024 tokens. Between each prefill chunk, the scheduler runs one decode step for waiting requests. The user with the 32K prompt gets slightly higher TTFT as chunks complete, but no single request holds the GPU for 1500ms straight.

Measured impact: ~68% reduction in p95 TTFT on 32K inputs in Sarathi-Serve benchmarks, without disaggregated hardware.

Without chunked prefill:
  32K prefill = 1500ms blocking → all decode requests see +1500ms queuing
  p95 TTFT at 50 concurrent users: 2000–3000ms

With chunked prefill (chunk=1024, 32 chunks):
  Each chunk: ~47ms GPU time
  32 chunks interleaved with decode steps
  p95 TTFT for other requests: ~100-300ms
  32K requester's TTFT: ~47ms × 32 = 1500ms (unchanged, but doesn't block others)

The trade-off: chunking slightly increases the total compute for very long prefills (minor overhead from chunk boundaries) and requires more scheduler complexity. Both are worth it.

Disaggregated prefill/decode: the second fix, at cost

Chunked prefill interleaves phases on the same GPU. Disaggregation takes the next step: separate GPU pools for each phase. Prefill requests go to high-FLOPS GPUs optimized for compute-bound work. After prefill completes, the KV cache is transferred to a decode GPU via high-speed interconnect (NVLink or RDMA), and decode runs on GPUs selected for HBM bandwidth.

vLLM's V1 engine (2025) implemented this via the NixlConnector, using NVIDIA's NIXL library for KV transfer. SGLang has its own router. The measured result on H100+H200 disaggregated pairs: ~2,100 tok/s versus ~1,200 tok/s colocated, a 75% throughput gain with lower cost-per-token despite the higher hourly spend.

Why does separating phases help even beyond chunked prefill?

• Prefill and decode compete for the same GPU resources when colocated. A long prefill starves decode even with chunking.
• Each GPU type can be right-sized: H100 SXM (989 TFLOPS) for prefill, H200 (4.8 TB/s HBM bandwidth) for decode.
• The decode pool can be scaled independently when output length increases (longer streaming responses → more decode work).

The complexity is real. You need: two GPU pools, KV transfer infrastructure, a disaggregation-aware scheduler, and the operational overhead of two fleets. The cost-per-token math only works at scale. For most teams at most traffic levels, chunked prefill is the right answer. Disaggregation is for when you've exhausted other options and are operating at hundreds of H100s.

flowchart LR
    RQ[Request] --> SCHED[Disaggregated scheduler]
    SCHED -->|"prompt tokens"| PF_GPU["Prefill GPU pool\nhigh FLOPS, H100 SXM"]
    PF_GPU -->|"KV cache transfer\n(NVLink / RDMA)"| DC_GPU["Decode GPU pool\nhigh HBM BW, H200"]
    DC_GPU -->|"stream tokens"| OUT[Response]

    style PF_GPU fill:#0e7490,color:#fff
    style DC_GPU fill:#0e7490,color:#fff
    style SCHED fill:#00e5ff,color:#0a0a0f

What breaks

Confusing TTFT problems with TPOT problems. You see high end-to-end latency. You add throughput capacity. It helps a little but not enough. You're fixing the wrong phase. Profile: check TTFT p50/p95 separately from TPOT p50/p95. Long TTFT → prefill problem (long prompts, no chunked prefill, GPU underprovisioned for compute). High TPOT → decode problem (batch too small, too little HBM bandwidth, quantization not applied).

Ignoring KV cache memory at deployment time. You benchmark a 70B model at batch=1 with 2K context and it fits in 80 GB. You deploy with 100 concurrent users and 16K context and hit OOM crashes within hours. The KV cache at 100 × 16K context for a 70B GQA model is: 100 × 16,000 × 0.32 MB = 512 GB — six times your GPU's HBM. Always calculate your concurrent-user KV memory budget before deploying. The KV cache article has the exact formula.

Benchmarking at batch=1 and calling it production-ready. Peak single-request throughput says nothing about p95 TTFT at 100 concurrent users. Real traffic has heavy-tailed prompt length distributions. A benchmark with average-length prompts overestimates throughput and misses the head-of-line blocking that degrades p95 by 10x.

Assuming newer GPU FLOPS matter for decode throughput. If your deploy is decode-bound (which it is for any interactive chatbot with short prompts and medium outputs), upgrading from A100 to H100 gives you the HBM bandwidth improvement (2.0 → 3.35 TB/s, ~67%) but not the full FLOPS improvement (312 → 989 TFLOPS, 3x). Budget accordingly.

Applying quantization only to weights. The KV cache is also BF16 by default. At long context, the KV cache bytes loaded per decode step can exceed the weight bytes. Switching KV cache dtype from BF16 to FP8 (supported in vLLM since 2024, near-zero quality delta on most workloads) can double the effective HBM bandwidth for decode. This is distinct from weight quantization.

The decision in practice

When you sit down to diagnose or design a serving system, start by measuring which phase is the bottleneck for your specific workload:

Your workload is prefill-heavy if: average prompt length is long (>4K tokens), TTFT SLA is tight, or you have RAG systems feeding large context chunks. Fix: chunked prefill (enable it in vLLM, it's the default in the V1 engine), raise chunk size to match your prompt distribution, consider disaggregated prefill if you're running at scale on H100+ hardware.

Your workload is decode-heavy if: average output length is long (>500 tokens), you're running code generation or summarization workloads, or TPOT is your binding constraint. Fix: continuous batching (already default in vLLM/SGLang), maximize batch size within your TTFT budget, apply quantization (FP8 on H100+ is a free throughput improvement, AWQ 4-bit for A100-class hardware).

Most production workloads are both. A customer-facing chatbot with 1K-token system prompts, 500-token user messages, and 300-token responses has non-trivial prefill and meaningful decode. Profile each independently. The throughput vs latency article has the full GPU economics of trading one against the other.

For serving engine choice: vLLM with the V1 engine gives you chunked prefill and continuous batching out of the box for most architectures. SGLang adds prefix caching via RadixAttention which directly reduces effective prefill work on requests that share prompt prefixes — critical for agent workloads with shared system prompts. Neither requires you to understand disaggregation unless you're at the scale where it matters. Start with vLLM defaults, profile, and tune chunk size and max batch tokens for your traffic shape. The serving engine comparison has the head-to-head benchmarks.

One thing that is not a fix: rewriting your application logic. Prefill and decode are properties of how transformers work, not bugs in your code. The optimizations exist at the serving layer, and understanding which phase you're fighting tells you exactly which lever to pull.