# Prefill vs Decode: The Two Phases of Every LLM Request > Why the first token takes longer than every other token, how compute-bound prefill and bandwidth-bound decode shape your entire serving architecture, and what to do about it. Canonical: https://ironclad.academy/ai-engineering/articles/prefill-vs-decode-phases | Difficulty: beginner | Topics: Inference, Serving, Latency, LLM Internals ## Key takeaways - Prefill is compute-bound (parallel matmul over all prompt tokens); decode is memory-bandwidth-bound (one token per step loading full weights + KV cache). The same GPU operates in fundamentally different regimes for each phase. - TTFT and TPOT are independent metrics driven by different phases — diagnosing latency requires measuring both separately, not just end-to-end latency. - Chunked prefill cuts p95 TTFT by ~68% on long prompts by interleaving prefill chunks with decode steps, eliminating head-of-line blocking without separate hardware. - GQA reduces KV cache memory 4-8x versus MHA by sharing KV heads across query head groups — the single most important architectural decision for long-context serving. - Disaggregated prefill/decode on H100+H200 pairs achieves ~75% higher throughput than colocated serving but adds significant operational complexity; only pursue it after exhausting chunked prefill and batching tuning. - Adding more CUDA cores does not help memory-bandwidth-bound decode. Only higher HBM bandwidth (newer GPU) or quantization (fewer bytes to load per step) moves the needle. > **In one line:** Every LLM request splits into two fundamentally different phases — compute-bound prefill that processes all prompt tokens at once, and memory-bandwidth-bound decode that generates one token per step — and this split determines your latency profile, memory budget, and serving architecture. **The idea.** When you send a request to an LLM, the GPU does two very different jobs. First, it runs a big parallel matrix multiplication over every token in your prompt simultaneously — that's prefill, and it takes however long it takes. Then, for each output token, it runs a much smaller forward pass that reads the entire model and growing context from GPU memory one token at a time — that's decode. The hardware arithmetic is so different between these two jobs that the bottleneck flips: compute for prefill, memory bandwidth for decode. Every serving optimization you'll read about in this section — [continuous batching](/ai-engineering/articles/continuous-batching-scheduling), [the KV cache](/ai-engineering/articles/kv-cache-gpu-memory-math), [speculative decoding](/ai-engineering/articles/speculative-decoding), [quantization](/ai-engineering/articles/quantization-gptq-awq-gguf-fp8) — only makes sense once you understand why these two phases behave so differently. **Key points.** - Prefill is compute-bound: the GPU is running large matrix multiplications over all prompt tokens at once, and FLOPS utilization is high. - Decode is memory-bandwidth-bound: each step loads all model weights and the KV cache from GPU HBM to generate one token. FLOPS utilization can be as low as 1-5%. - TTFT (time to first token) is a prefill metric. TPOT (time per output token) is a decode metric. They respond to different optimizations. - GQA shrinks the KV cache 4-8x compared to MHA, making long-context decode tractable. - Chunked prefill eliminates head-of-line blocking on long prompts. Disaggregated prefill/decode runs each phase on separate GPU pools for maximum throughput at scale. **Back-of-the-envelope: prefill vs decode compute intensity.** ``` Llama 3.1 8B: ~8B parameters, BF16 → 16 GB weights Prefill (1024-token prompt): Dominant op: QKV projection → O(T × d_model × d_head × num_heads) = large matmul, high arithmetic intensity → GPU cores busy Decode (one token): Same weights, but now seq_len=1 Weight load: ~16 GB per step (full weights + KV) Compute: ~16 GFLOPs per token H100 SXM HBM bandwidth: 3.35 TB/s → Ceiling: 3,350 GB/s ÷ 16 GB = ~209 steps/s peak (single request) At that throughput: ~4.8ms per token, 0.2% of H100's 989 TFLOPS utilized → Bandwidth-bound, not compute-bound ``` ```mermaid flowchart LR REQ[Incoming request] --> PF[Prefill phase] PF -->|"processes all N tokens\nin one parallel pass"| KV[KV cache written] KV --> DC[Decode phase] DC -->|"generates 1 token\nper iteration"| TOK[Token emitted] TOK -->|"append to KV cache,\nrepeat"| DC TOK -->|EOS token| RESP[Response complete] style PF fill:#0e7490,color:#fff style DC fill:#0e7490,color:#fff style KV fill:#15803d,color:#fff ``` **Key design decisions.** | Decision | Prefill concern | Decode concern | | --- | --- | --- | | Batch size | Larger batches increase compute utilization | Larger batches increase memory pressure and queuing latency | | Attention variant | MHA, MQA, GQA all look similar at prefill | GQA dramatically reduces KV bytes loaded per decode step | | GPU selection | More FLOPS → faster prefill | More HBM bandwidth → faster decode | | Quantization | Minor effect on prefill speed | Reduces bytes loaded per step, directly improves decode throughput | | Chunked prefill | Slices long prefills to avoid blocking decode | Decode requests interleave with prefill chunks | **If you have 60 seconds, say this.** "Every LLM request has two phases. Prefill runs once: the GPU processes all your prompt tokens in parallel — it's a big matrix multiplication and it's compute-bound. Decode runs once per output token: the GPU loads the entire model and KV cache from memory to produce each token — it's memory-bandwidth-bound. Time to first token is a prefill problem. Streaming speed is a decode problem. They need different fixes. On a single H100, decode for a lone request uses maybe 1-5% of available FLOPS because you're waiting on memory loads, not math. That's why batching helps — you amortize those memory loads across more tokens — and why quantization helps decode more than prefill — fewer bytes to load per step." Your production chatbot has a 600ms pause before it streams any text. You profile the server and find your request spent most of that window queued behind someone else's 30K-token prompt — which pinned the GPU at full compute utilization the entire time. Meanwhile, your p50 inter-token latency is fine. You're staring at the canonical prefill problem: head-of-line blocking. Adding GPUs buys capacity in proportion to spend, with no efficiency win — the pause just gets rarer. The actual fix is scheduling, not "buy more compute." Understanding why requires understanding what the GPU is actually doing during each phase of an LLM request. ## What happens during prefill When a request arrives at a serving engine, the first job is to process the prompt. Suppose the prompt is 2,048 tokens. The model needs to compute key-value pairs for every one of those tokens across every attention layer, and produce the hidden state that will be used to predict the first output token. The critical property is that this happens in parallel. All 2,048 tokens are processed simultaneously in one forward pass. The dominant operation is matrix multiplication: the 2,048-token sequence gets projected through the model's weight matrices in large, dense matmuls. On modern GPUs, large dense matmuls are exactly the operation for which the hardware is designed — the arithmetic intensity (FLOPS per byte of memory accessed) is high, and the GPU's tensor cores stay busy. ```mermaid sequenceDiagram participant C as Client participant S as Serving Engine participant G as GPU C->>S: POST /v1/completions (prompt: 2048 tokens) S->>G: prefill pass: process tokens 1..2048 simultaneously Note over G: Large dense matmuls, high FLOP utilization G->>S: KV cache written for all 2048 tokens G->>S: first output token logits S->>C: first token (TTFT = time so far) loop decode: one token per iteration G->>S: load weights + KV cache from HBM G->>S: next token logits S->>C: stream token end ``` **TTFT** — time to first token — is entirely prefill time plus queueing delay. A 4K-token prompt might take 50–200ms to prefill on an A100; a 32K-token prompt can take 500–2000ms. Users experience this as the "thinking pause" before any text appears. ## What happens during decode After prefill, the model enters decode: it generates one token per step, feeds that token back as input, and repeats until it hits an EOS token or a maximum length. Each decode step is a forward pass with a sequence of length one — the single new token — plus the KV cache from all previous tokens. Here's where the arithmetic flips. A single-token forward pass does almost no compute relative to the data it has to read. To generate one token, the GPU must: 1. Load all model weights from HBM — that's 16 GB for an 8B BF16 model, or 140 GB for a 70B model. 2. Load the KV cache for every previous token in the context — growing by one entry per step. 3. Run the forward pass — which, for a single token, is mathematically small. The ratio of compute to memory bandwidth required is extremely low. This is the definition of **memory-bandwidth-bound**: the GPU has enormous math capacity sitting idle while it waits for data to arrive from HBM. An H100 SXM can do 989 TFLOPS of BF16 math. During single-request decode, you might use 5–20 TFLOPS of that — because the bottleneck is the 3.35 TB/s HBM bandwidth ceiling, not the math units. ``` Decode ceiling: 70B BF16, tensor-parallel across 2× H100 SXM (140 GB of weights doesn't fit one 80 GB card — TP2 is the minimum) Weight load per step: ~140 GB total, ~70 GB per GPU Aggregate HBM bandwidth: 2 × 3.35 TB/s = 6.7 TB/s Max steps/sec: 6,700 / 140 ≈ 48 tok/s per request Real-world TP2 decode: ~30-40 tok/s at batch=1 (kernel launch + all-reduce overhead eats the rest) FLOP utilization at 40 tok/s: ~140 GFLOPs / step × 40 steps/s = ~5.6 TFLOPS → 5.6 / 1,978 aggregate ≈ 0.3% of available FLOPS ``` This matters because most instincts about "my GPU is slow" lead to buying more FLOPS. But for decode, more FLOPS do not help. Only two things move the needle: higher HBM bandwidth (which is why the H100 → H200 → B200 progression matters for serving) or quantization (which reduces the bytes you load per step). ## The two metrics you actually care about **TTFT** and **TPOT** (time per output token, also called TBT — time between tokens) measure different user experiences and respond to different fixes. | Metric | Phase | User experience | What moves it | | --- | --- | --- | --- | | TTFT | Prefill | "Why is it thinking so long?" | Prompt length, chunked prefill, disaggregated prefill, GPU FLOPS | | TPOT / TBT | Decode | "Why is the text streaming slowly?" | Batch size, HBM bandwidth, KV cache size, quantization | | End-to-end latency | Both | Total time for complete response | Dominated by decode for long outputs | A common mistake: a team sees high end-to-end latency on their API, adds capacity, and wonders why it didn't help proportionally. Because the response was 2,000 tokens and the bottleneck was decode throughput, not prefill latency. Measuring only one number hides which phase needs work. [Interactive visualizer on the original page: batching — How batching shifts the decode bottleneck] ## The KV cache: prefill writes it, decode reads it When the model processes the prompt during prefill, it computes key and value tensors for each attention head at each layer. These get written to a **KV cache** in GPU memory and reused on every subsequent decode step — rather than recomputing attention over the full history from scratch each time. The KV cache is not free. Per-token KV memory scales as: ``` KV bytes per token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element Llama 3.1 8B (GQA, 8 KV heads, 32 layers, head_dim=128, BF16): = 2 × 32 × 8 × 128 × 2 = 131,072 bytes ≈ 0.125 MB/token At 100 concurrent requests × 4K context: = 100 × 4,000 × 0.125 MB = 50 GB KV cache + 16 GB model weights = 66 GB total → fits an H100 80GB, barely Same workload with 32K context: = 100 × 32,000 × 0.125 MB = 400 GB → does not fit ``` This is why **grouped-query attention (GQA)** matters. MHA (multi-head attention) uses the same number of KV heads as query heads. GQA groups multiple query heads to share one KV head. Llama 3.1 70B uses 64 query heads but only 8 KV heads — an 8x reduction in KV cache memory, directly translating to either 8x more concurrent requests or 8x longer context for the same GPU. The [KV cache article](/ai-engineering/articles/kv-cache-gpu-memory-math) has the full memory math and PagedAttention — which solves KV cache fragmentation. Understand that article before you size a production deployment. [Interactive visualizer on the original page: kv-cache — KV cache VRAM vs model weights] ## What limits throughput in each phase The reason these phases need different optimizations becomes concrete when you look at what is actually saturated. ```mermaid flowchart TD PF_BOUND["Prefill: compute-bound"] DC_BOUND["Decode: memory-bandwidth-bound"] PF_BOUND --> PF1["FLOPS determine speed"] PF_BOUND --> PF2["Bigger matmuls = better utilization"] PF_BOUND --> PF3["More concurrent prompts helps little"] PF_BOUND --> PF4["Fix: chunked prefill, faster GPU, flash attention"] DC_BOUND --> DC1["HBM bandwidth determines speed"] DC_BOUND --> DC2["Loading weights dominates time"] DC_BOUND --> DC3["Batching: amortize weight loads across requests"] DC_BOUND --> DC4["Fix: quantization, higher-bandwidth GPU, batching"] style PF_BOUND fill:#0e7490,color:#fff style DC_BOUND fill:#0e7490,color:#fff ``` **Batching works for decode specifically because of memory bandwidth.** If you serve 32 requests simultaneously during decode instead of 1, you still load the weights roughly once per step and run 32 token computations with that data. The memory load cost is amortized across 32 outputs. This is why continuous batching (covered in depth in the [continuous batching article](/ai-engineering/articles/continuous-batching-scheduling)) achieves 3-24x throughput improvement over static batching — it keeps GPU batch slots filled. **Quantization improves decode more than prefill.** Going from BF16 to FP8 halves the bytes loaded per decode step, roughly doubling decode throughput ceiling. The effect on prefill is smaller because prefill is already compute-bound — you're not waiting on memory loads. The [quantization article](/ai-engineering/articles/quantization-gptq-awq-gguf-fp8) covers GPTQ, AWQ, and FP8 in detail. ## Chunked prefill: the first fix for TTFT at scale Without any special handling, a 32K-token prefill monopolizes the GPU for potentially 500-1500ms. Every other request in the queue waits. This is head-of-line blocking, and it destroys p95 TTFT for all requests whenever a single long prompt arrives. **Chunked prefill** (Sarathi-Serve, now default in vLLM) slices the prefill into fixed-size chunks — typically 512 or 1024 tokens — and interleaves them with decode steps from other in-flight requests. A 32K prompt becomes 32 chunks of 1024 tokens. Between each prefill chunk, the scheduler runs one decode step for waiting requests. The user with the 32K prompt gets slightly higher TTFT as chunks complete, but no single request holds the GPU for 1500ms straight. Measured impact: **~68% reduction in p95 TTFT** on 32K inputs in Sarathi-Serve benchmarks, without disaggregated hardware. ``` Without chunked prefill: 32K prefill = 1500ms blocking → all decode requests see +1500ms queuing p95 TTFT at 50 concurrent users: 2000–3000ms With chunked prefill (chunk=1024, 32 chunks): Each chunk: ~47ms GPU time 32 chunks interleaved with decode steps p95 TTFT for other requests: ~100-300ms 32K requester's TTFT: ~47ms × 32 = 1500ms (unchanged, but doesn't block others) ``` The trade-off: chunking slightly increases the total compute for very long prefills (minor overhead from chunk boundaries) and requires more scheduler complexity. Both are worth it. ## Disaggregated prefill/decode: the second fix, at cost Chunked prefill interleaves phases on the same GPU. Disaggregation takes the next step: separate GPU pools for each phase. Prefill requests go to high-FLOPS GPUs optimized for compute-bound work. After prefill completes, the KV cache is transferred to a decode GPU via high-speed interconnect (NVLink or RDMA), and decode runs on GPUs selected for HBM bandwidth. vLLM's V1 engine (2025) implemented this via the NixlConnector, using NVIDIA's NIXL library for KV transfer. SGLang has its own router. The measured result on H100+H200 disaggregated pairs: **~2,100 tok/s versus ~1,200 tok/s colocated**, a 75% throughput gain with lower cost-per-token despite the higher hourly spend. Why does separating phases help even beyond chunked prefill? - Prefill and decode compete for the same GPU resources when colocated. A long prefill starves decode even with chunking. - Each GPU type can be right-sized: H100 SXM (989 TFLOPS) for prefill, H200 (4.8 TB/s HBM bandwidth) for decode. - The decode pool can be scaled independently when output length increases (longer streaming responses → more decode work). The complexity is real. You need: two GPU pools, KV transfer infrastructure, a disaggregation-aware scheduler, and the operational overhead of two fleets. The cost-per-token math only works at scale. For most teams at most traffic levels, chunked prefill is the right answer. Disaggregation is for when you've exhausted other options and are operating at hundreds of H100s. ```mermaid flowchart LR RQ[Request] --> SCHED[Disaggregated scheduler] SCHED -->|"prompt tokens"| PF_GPU["Prefill GPU pool\nhigh FLOPS, H100 SXM"] PF_GPU -->|"KV cache transfer\n(NVLink / RDMA)"| DC_GPU["Decode GPU pool\nhigh HBM BW, H200"] DC_GPU -->|"stream tokens"| OUT[Response] style PF_GPU fill:#0e7490,color:#fff style DC_GPU fill:#0e7490,color:#fff style SCHED fill:#00e5ff,color:#0a0a0f ``` ## What breaks **Confusing TTFT problems with TPOT problems.** You see high end-to-end latency. You add throughput capacity. It helps a little but not enough. You're fixing the wrong phase. Profile: check TTFT p50/p95 separately from TPOT p50/p95. Long TTFT → prefill problem (long prompts, no chunked prefill, GPU underprovisioned for compute). High TPOT → decode problem (batch too small, too little HBM bandwidth, quantization not applied). **Ignoring KV cache memory at deployment time.** You benchmark a 70B model at batch=1 with 2K context and it fits in 80 GB. You deploy with 100 concurrent users and 16K context and hit OOM crashes within hours. The KV cache at 100 × 16K context for a 70B GQA model is: 100 × 16,000 × 0.32 MB = 512 GB — six times your GPU's HBM. Always calculate your concurrent-user KV memory budget before deploying. The [KV cache article](/ai-engineering/articles/kv-cache-gpu-memory-math) has the exact formula. **Benchmarking at batch=1 and calling it production-ready.** Peak single-request throughput says nothing about p95 TTFT at 100 concurrent users. Real traffic has heavy-tailed prompt length distributions. A benchmark with average-length prompts overestimates throughput and misses the head-of-line blocking that degrades p95 by 10x. **Assuming newer GPU FLOPS matter for decode throughput.** If your deploy is decode-bound (which it is for any interactive chatbot with short prompts and medium outputs), upgrading from A100 to H100 gives you the HBM bandwidth improvement (2.0 → 3.35 TB/s, ~67%) but not the full FLOPS improvement (312 → 989 TFLOPS, 3x). Budget accordingly. **Applying quantization only to weights.** The KV cache is also BF16 by default. At long context, the KV cache bytes loaded per decode step can exceed the weight bytes. Switching KV cache dtype from BF16 to FP8 (supported in vLLM since 2024, near-zero quality delta on most workloads) can double the effective HBM bandwidth for decode. This is distinct from weight quantization. ## The decision in practice When you sit down to diagnose or design a serving system, start by measuring which phase is the bottleneck for your specific workload: **Your workload is prefill-heavy if:** average prompt length is long (>4K tokens), TTFT SLA is tight, or you have RAG systems feeding large context chunks. Fix: chunked prefill (enable it in vLLM, it's the default in the V1 engine), raise chunk size to match your prompt distribution, consider disaggregated prefill if you're running at scale on H100+ hardware. **Your workload is decode-heavy if:** average output length is long (>500 tokens), you're running code generation or summarization workloads, or TPOT is your binding constraint. Fix: continuous batching (already default in vLLM/SGLang), maximize batch size within your TTFT budget, apply quantization (FP8 on H100+ is a free throughput improvement, AWQ 4-bit for A100-class hardware). **Most production workloads are both.** A customer-facing chatbot with 1K-token system prompts, 500-token user messages, and 300-token responses has non-trivial prefill and meaningful decode. Profile each independently. The [throughput vs latency article](/ai-engineering/articles/throughput-vs-latency-gpu-economics) has the full GPU economics of trading one against the other. For serving engine choice: vLLM with the V1 engine gives you chunked prefill and continuous batching out of the box for most architectures. SGLang adds prefix caching via RadixAttention which directly reduces effective prefill work on requests that share prompt prefixes — critical for agent workloads with shared system prompts. Neither requires you to understand disaggregation unless you're at the scale where it matters. Start with vLLM defaults, profile, and tune chunk size and max batch tokens for your traffic shape. The [serving engine comparison](/ai-engineering/articles/serving-engine-comparison-2026) has the head-to-head benchmarks. One thing that is not a fix: rewriting your application logic. Prefill and decode are properties of how transformers work, not bugs in your code. The optimizations exist at the serving layer, and understanding which phase you're fighting tells you exactly which lever to pull. ## Frequently asked questions Q: What is the difference between prefill and decode in LLM inference? A: Prefill processes all prompt tokens in one parallel forward pass — a large matrix multiplication that is compute-bound and typically takes 20–500ms for a long prompt. Decode generates one token per step, loading the full model weights and growing KV cache from GPU memory on each iteration — making it memory-bandwidth-bound. The same GPU behaves differently in each phase, and optimizing one often makes the other worse. Q: What is TTFT and why is it different from TPOT? A: TTFT (time to first token) measures how long the user waits before seeing any output — it is dominated by the prefill phase. TPOT (time per output token, also called TBT or inter-token latency) measures how fast tokens stream after the first one — it is dominated by the decode phase. A chatbot user notices TTFT as the "thinking pause"; streaming responsiveness is driven by TPOT. Production SLAs should track both independently because fixing one does not fix the other. Q: Why is decode memory-bandwidth-bound while prefill is compute-bound? A: During decode the model generates one token at a time. That single token triggers a full forward pass that loads every model weight and KV cache entry from GPU HBM — but performs very few actual multiply-accumulate operations per byte loaded, so the bottleneck is HBM read bandwidth, not FLOPS. Prefill processes hundreds or thousands of tokens simultaneously in a single large matrix multiplication that keeps GPU cores busy — so the bottleneck is compute. This is the arithmetic intensity difference: high for prefill, extremely low for decode. Q: What is chunked prefill and why does it help? A: Chunked prefill (also called Sarathi-Serve, now default in vLLM) splits a long prompt into fixed-size chunks — typically 512 or 1024 tokens — and interleaves prefill chunks with decode steps from other requests. Without chunking, a 32K-token prompt monopolizes the GPU for hundreds of milliseconds, causing every other request to wait (head-of-line blocking). Chunking cuts p95 TTFT by ~68% on long inputs in measured benchmarks without requiring separate hardware for prefill and decode. Q: What is disaggregated prefill/decode and when is it worth the complexity? A: Disaggregated prefill/decode routes each phase to a dedicated GPU pool — prefill-optimized GPUs (high FLOPS) handle prompt processing, decode-optimized GPUs (high HBM bandwidth) handle token generation. The KV cache is transferred between pools after prefill completes. On H100+H200 pairs, disaggregation achieves roughly 2,100 tok/s versus ~1,200 colocated — a 75% throughput gain. The complexity is real: KV transfer latency must be hidden, and you need two GPU pools. Only justified at scale where the cost-per-token math makes it worthwhile. Q: How does GQA reduce KV cache memory in the decode phase? A: Multi-head attention (MHA) maintains one KV pair per attention head. Grouped-query attention (GQA) groups multiple query heads to share a single KV head — Llama 3.1 70B uses 8 KV heads instead of 64 query heads, an 8x reduction in KV cache size. At long context lengths the KV cache can dwarf model weights, so GQA is the primary architectural lever for making large-context serving tractable. Multi-query attention (MQA) is the extreme version: a single KV head shared by all query heads.