Throughput vs Latency: The GPU Economics of LLM Serving

A team I spoke with had just launched a legal document summarization service. Their benchmark said ~800 tokens/sec, latency was fast in testing. Three weeks after launch, legal teams were complaining about 12-second waits for the first word of the summary. Nothing had changed. Their benchmark used batch=1. In production they had 40 concurrent users. The GPU was spending 60% of its time idle, waiting for the batch to fill and the longest document to finish before it would release results to anyone. The fix — enabling continuous batching and tuning the max batch token budget — took four hours and halved their monthly GPU bill as a side effect.

That's the fundamental issue with LLM serving economics: most engineers measure the wrong thing and optimize in the wrong direction. This article maps the actual levers.

The two bottlenecks you are always fighting

LLM inference has two distinct execution phases with completely different hardware characteristics, and they respond to different interventions. The prefill and decode phases article covers the mechanism in depth; what matters here is the economic consequence.

Prefill is compute-bound. Processing the prompt runs large matrix multiplications over all input tokens simultaneously. The GPU is doing real work — FLOP utilization is high. Making prefill faster means more CUDA cores, tensor parallelism, or reducing the prefill compute load (chunked prefill, prompt caching).

Decode is memory-bandwidth-bound. Generating each output token requires loading the full model weights plus the entire KV cache from HBM (high-bandwidth memory) to compute a single matrix-vector multiplication. The arithmetic intensity — FLOPS executed per byte moved — is around 1 for decode at typical batch sizes, versus ~1,000 on the hardware roofline. The GPU's CUDA cores sit mostly idle waiting for data.

This means the key hardware figure for decode speed is not peak TFLOPS but HBM bandwidth:

A100 80GB SXM:   2.0 TB/s HBM2e,   312 TFLOPS BF16
H100 80GB SXM:   3.35 TB/s HBM3,   989 TFLOPS BF16
B200 180GB SXM:  ~8.0 TB/s HBM3e,  ~4.5 PFLOPS BF16

Decode throughput ratio H100 vs A100:
  bandwidth ratio = 3.35 / 2.0 ≈ 1.67
  → H100 delivers ~67% more decode tokens/sec than A100
  The 3× TFLOPS advantage is irrelevant for bandwidth-bound decode.

This is why quantization actually speeds up inference beyond just fitting the model on smaller hardware. FP8 halves the bytes per parameter compared to BF16, which means the GPU can load twice as many model weights per second from HBM. At the same HBM bandwidth, you get roughly twice the decode throughput. The CUDA cores were never the constraint.

flowchart TD
    DEC["Decode step\n(one token)"] --> LOAD["Load model weights\n+ KV cache from HBM"]
    LOAD --> MATMUL["Single matrix-vector multiply\n(low arithmetic intensity)"]
    MATMUL --> TOKEN["Sample next token"]
    TOKEN --> DEC
    BW["HBM bandwidth limit"] -.->|bottleneck| LOAD
    style BW fill:#ff2e88,color:#111
    style LOAD fill:#ffaa00,color:#0a0a0f

The batch size dial

Batch size is the single knob that controls where you sit on the throughput-latency curve. Understanding why requires seeing what happens inside the GPU at different batch sizes.

At batch=1, the GPU runs one request at a time. Compute utilization is low because there aren't enough parallel operations to fill the tensor cores. Each token step loads the entire model weights from HBM, does almost nothing with them, and discards them. This is the fastest path to first token for that one user — but the GPU is largely idle, and cost per token is high.

At batch=64, 64 sequences generate tokens simultaneously. The matrix multiplications are now proper matrix-matrix operations, not matrix-vector. Tensor core utilization climbs. You generate 64× more tokens per step from the same weight-load cost. Cost per token drops proportionally. But every one of those 64 users is waiting longer in queue before their request can start, and TTFT rises.

The practical operating point for most production systems is the largest batch size where TTFT p95 stays under your SLA. That's it. Everything else is optimization within that constraint.

{
  "type": "batching",
  "title": "Static vs Continuous Batching: GPU Utilization and Throughput",
  "mode": "static"
}

Continuous batching changed everything

Static batching — where you fill a batch to capacity, run the entire batch until the longest sequence finishes, then start the next batch — was the default before 2023. It has a brutal failure mode: a single 4,000-token output in a batch of requests with 100-token outputs holds the entire batch hostage. The short requests finish their computation in 20 steps and then idle for 180 more steps while the GPU waits on the long one. GPU utilization at mixed-length workloads was typically 20-40%.

Continuous batching (also called in-flight batching or iteration-level scheduling) fixes this by treating the batch as a rolling slot pool. When any sequence finishes generating a token that terminates it, the scheduler immediately fills that slot with a waiting request. The GPU is always doing work. Anyscale's original continuous-batching benchmark (OPT-13B, mixed-length traffic) measured up to ~23× higher throughput versus static batching at high concurrency; a more typical production gain is 3-4×, with the multiple growing as length variance makes static batching collapse.

This is why continuous batching was the most impactful serving optimization of 2023-2024. It required no model changes, no new hardware, and no quality tradeoffs. It just stopped throwing away half the GPU's cycles.

vLLM's V1 engine (2025) extended this with async scheduling — the scheduler runs off the critical path, overlapping CPU scheduling decisions with GPU execution — and disaggregated prefill support via NVIDIA's NIXL library.

Cost per token: the actual formula

"We're running on H100s, so we should be efficient" is not a cost model. Here is one:

System throughput (tok/s):  T
GPU count:                  G
GPU hourly rate:            R  ($ per GPU-hour)

Cost per million output tokens = (G × R) / (T × 3,600 / 1,000,000)
                               = (G × R) / (T × 0.0036)

Example configurations (illustrative, mid-2026 spot rates, Llama 3.1 8B-class model):

Config 1: 1× H100, static batching, batch≈8
  T ≈ 400 tok/s, R = $4.00/hr, G = 1
  Cost = $4.00 / (400 × 0.0036) = $4.00 / 1.44 = $2.78/M tokens

Config 2: 1× H100, continuous batching, BF16, batch≈64
  T ≈ 1,800 tok/s, R = $4.00/hr, G = 1
  Cost = $4.00 / (1,800 × 0.0036) = $4.00 / 6.48 = $0.62/M tokens

Config 3: 1× H100, continuous batching, FP8, batch≈64
  T ≈ 2,800 tok/s, R = $4.00/hr, G = 1
  Cost = $4.00 / (2,800 × 0.0036) = $4.00 / 10.08 = $0.40/M tokens

Config 4: 2× H100 disaggregated prefill/decode, FP8
  T ≈ 4,200 tok/s, R = $4.00/hr, G = 2
  Cost = $8.00 / (4,200 × 0.0036) = $8.00 / 15.12 = $0.53/M tokens
  [More throughput than config 3, but 2 GPUs — cost/token is higher here
   unless you're prefill-latency-constrained, in which case it's worth it]

Config 2 versus Config 1 is free money — same hardware, same model, same SLA, 4.5× lower cost. Config 3 costs 35% less than Config 2 with near-zero quality change on most tasks (typically 1-2% MMLU delta for AWQ INT4 or FP8). Config 4 costs more per token than Config 3 but solves a different problem: prefill latency at high concurrency.

The punchline is that algorithmic improvements (continuous batching, quantization, prefix caching) reliably outperform hardware upgrades on a cost-per-token basis until you've exhausted them.

The four regimes and what works in each

flowchart LR
    R1["Prefill-bound\n(long prompts, low concurrency)"] -->|fix| F1["Chunked prefill\nPrompt caching\nPrefill disaggregation"]
    R2["Decode-bound / latency\n(short prompts, low concurrency)"] -->|fix| F2["Speculative decoding\nQuantization\nSmaller model"]
    R3["Decode-bound / throughput\n(high concurrency)"] -->|fix| F3["Continuous batching\nQuantization\nGQA / MQA"]
    R4["Memory-limited\n(too many parallel contexts)"] -->|fix| F4["PagedAttention\nKV cache quantization\nLimit max context"]
    style R1 fill:#0e7490,color:#fff
    style R2 fill:#a855f7,color:#fff
    style R3 fill:#15803d,color:#fff
    style R4 fill:#00e5ff,color:#0a0a0f

Prefill-bound. Symptoms: high TTFT, especially for long prompts. The prefill of a 32K-token document genuinely takes time — attention compute grows with the square of prompt length (FlashAttention removes the quadratic memory cost, not the quadratic compute), though prefill looks near-linear at moderate lengths because MLP FLOPs dominate until the attention term takes over. Fixes: prompt caching (skip prefill entirely for repeated prefixes), prefill disaggregation (dedicated prefill GPU does the compute, hands KV cache to decode GPU), and — when long prompts share the GPU with live traffic — chunked prefill, which slices the long prefill into chunks interleaved with decode. Chunked prefill doesn't make the long request itself faster; it stops that request from stalling every other in-flight request's TTFT and token cadence.

Decode-bound, latency-sensitive. Symptoms: TTFT is fine but TPOT is slow; individual users are watching tokens trickle out. This is the speculative decoding sweet spot. A small draft model (or self-speculative heads via EAGLE-2) proposes K tokens; the target model verifies all K in one forward pass. At acceptance rate α=0.7 with K=5, expected tokens per step = 1 + Σαⁱ ≈ 2.9, so you're getting ~2.9 tokens for one target forward pass plus the draft model's overhead (typically ~10-20% of a target step). The result is 2-3× faster TPOT at low batch sizes. This is covered in depth at speculative decoding.

Decode-bound, throughput-sensitive. Symptoms: high concurrency, GPU utilization looks good, but cost per token is higher than expected. This is the continuous batching + quantization regime. The KV cache memory math article has the detailed formula; the short version is that quantization to FP8 doubles the effective HBM bandwidth for weight loading, and GQA reduces the KV cache size by 4-8× versus vanilla MHA, letting you run larger batches without running out of memory.

Memory-limited. Symptoms: OOM crashes or severe KV cache eviction under moderate concurrency, even when GPU compute utilization seems reasonable. The culprit is usually a mismatch between expected context length and actual traffic. At 100 concurrent requests × 8K context each, a 70B model in BF16 (GQA, ~0.31 MB of KV per token) needs roughly 250 GB just for the KV cache on top of the 140 GB for weights — a multi-GPU problem before the first token is generated, so the real fixes are reducing concurrency per replica, shrinking max_model_len, or quantizing the KV cache. PagedAttention (default in vLLM) handles fragmentation, but it doesn't create memory that isn't there.

Speculative decoding: when the math works and when it doesn't

Speculative decoding is often presented as a free latency speedup. It is not. The speedup depends on the acceptance rate α — how often the target model agrees with the draft model's token proposals — and the batch size.

Expected tokens per step = 1 + α + α² + ... + α^K  (geometric series for K draft tokens)

At K=5:
  α = 0.9  → expected 4.7 tokens/step  (target agrees almost always: common text)
  α = 0.7  → expected 2.9 tokens/step  (target agrees 70%: typical code generation)
  α = 0.5  → expected 2.0 tokens/step  (target overrides often: creative/uncertain tasks)
  α = 0.3  → expected 1.4 tokens/step  (barely worth the draft overhead)

The batch size constraint is just as important. At batch=1 or batch=2, the target model's forward pass has headroom — verifying K tokens in parallel costs about the same GPU time as verifying 1. The speedup is real. At batch=64, the target model's forward pass is already compute-saturated; adding K verification tokens per step increases compute load proportionally with no free lunch. Speculative decoding's gain disappears or inverts at high batch sizes.

EAGLE-2 (the 2025 self-speculative variant now default in SGLang) sidesteps the separate draft model problem by attaching auxiliary prediction heads to the target model itself. These share the target's hidden states, so they're both cheaper to run and tend to have higher acceptance rates than a fully separate draft model. The tradeoff: they require a training step to attach the heads to your specific model.

One pitfall: over-quantizing the draft model to INT4 in search of a cheaper draft pass. The acceptance rate tanks, the overhead of running more rejected proposals accumulates, and the net latency can be worse than no speculative decoding. Keep the draft model at FP16 or INT8.

{
  "type": "speculative",
  "title": "Speculative Decoding: Acceptance Rate vs Speedup",
  "acceptance": 0.7
}

Prefill-decode disaggregation: the advanced play

In a colocated serving setup, prefill and decode compete for the same GPU. A long prefill monopolizes the GPU while all concurrent decode sequences stall — this is the primary source of TTFT spikes in production under mixed-length traffic.

Disaggregated serving puts prefill on dedicated GPUs ("prefill workers") and decode on separate GPUs ("decode workers"). The prefill GPU processes the prompt, generates the KV cache, and transfers it to a decode GPU via NVLink or InfiniBand. The decode GPU handles generation without any compute interruption from new arrivals.

Measured results on H100+H200 pairs: ~2,100 tok/s versus ~1,200 tok/s colocated — a 75% throughput gain. Note what that means for cost: 2× the hardware for 1.75× the tokens, so cost per token rises ~15% at these numbers (this is the Config 4 math above). What you're buying is p95 TTFT isolation; disaggregation pays for itself only when prefill latency is the binding constraint.

The catch is operational complexity. You need:

• Dedicated hardware pairs (or dynamic GPU partitioning)
• High-bandwidth interconnects between prefill and decode workers
• KV cache transfer serialization (NIXL in vLLM V1, native support in SGLang via mooncake-style transfer)
• Separate scaling policies for the two GPU pools

For deployments below ~10 H100s at sustained load, this complexity is not justified. Chunked prefill (which interleaves prefill chunks with decode steps on the same GPU) gets most of the TTFT benefit without disaggregated hardware. vLLM makes chunked prefill the default; it's the right starting point.

What breaks

Benchmarking at the wrong concurrency. Measuring throughput at batch=1 and assuming it represents production capacity is the most common mistake in this space. A serving stack that achieves 3,000 tok/s at batch=1 might achieve 2,000 tok/s at 100 concurrent users — or might crash with OOM because no one calculated KV cache memory at realistic concurrency. Benchmark at 100-200 concurrent users with realistic prompt/output length distributions before calling a deployment production-ready.

Heavy-tailed length distributions. Real traffic is not uniform. A customer service bot might have 80% of requests under 500 tokens, but 5% exceed 16K tokens. Those long requests will dominate p99 TTFT and can starve out shorter requests under static scheduling. Continuous batching with priority scheduling (or preemption) handles this, but only if you've measured the tail of your length distribution first.

KV cache memory underestimation. Engineers often calculate VRAM as "model weights + some buffer." At 100 concurrent requests × 4K average context, a 7B model (GQA, BF16) needs roughly 14 GB for weights — but the KV cache adds another ~50 GB (0.125 MB/token × 4,096 tokens × 100 requests). That exceeds the model weights by 3×. At 8K context it doubles again. Scale to a 70B model and the KV cache grows ~2.5× per token while the weights grow 10×. The KV cache memory math article has the exact formula; run it before deploying, not after the first OOM.

Speculative decoding at high batch sizes. Teams that benchmark speculative decoding at batch=1 (seeing 2.5× speedup) and deploy it in a throughput-optimized setup (batch=64) are often disappointed. Monitor acceptance rate and actual tokens/step continuously, not just at deployment time. If the workload shifts to more creative or uncertain outputs, acceptance rate drops and speculative decoding starts costing more than it saves.

Ignoring TGI. As of December 2025, Hugging Face's Text Generation Inference entered maintenance mode. HuggingFace itself now directs users to vLLM or SGLang. Teams still running TGI in production aren't leaving 24× on the table — TGI has had continuous batching since 2023 — but engine-swap gains of 1.2-2× are common depending on workload, and a stack in maintenance mode won't pick up the next generation of optimizations: EAGLE-class speculative decoding, disaggregation, new attention kernels. Migrating amounts to a swap of the serving stack — same model, same weights, same hardware.

Default chunk sizes. vLLM's chunked prefill chunk size (--max-num-batched-tokens), SGLang's --chunked-prefill-size, and the KV block size all have defaults that are calibrated for typical workloads. If your traffic has a distinct shape — very long prompts, very short outputs, massive batch sizes, or heavy prefix reuse — these defaults may leave 20-40% performance on the table. Profiling your actual workload and tuning these parameters is not optional at production scale.

The decision in practice

The optimization sequence that works across most deployments:

1. Enable continuous batching. If you're on a framework that doesn't have it, switch. This is always step one and recovers the most headroom.

2. Calculate your KV memory budget. Use the formula from KV cache memory math. Set max_model_len to what you actually need, not the model's maximum. Every unused context slot is KV cache memory that could be serving more requests.

3. Apply quantization. FP8 on H100/H200 (hardware-native, near-BF16 quality), AWQ INT4 on older hardware or for large models that don't fit in BF16. This is the second-biggest lever and costs almost nothing in quality for most tasks.

4. Measure at production concurrency. p95 TTFT and TPOT at 100-200 concurrent users. Not batch=1, not average latency. This is where your SLA either holds or breaks.

5. If TTFT is the problem: Enable chunked prefill. If you have the hardware budget, explore prefill-decode disaggregation. If specific prompts are repeated heavily, enable prefix caching (automatic in vLLM v0.4+, RadixAttention in SGLang).

6. If per-request TPOT is the problem: Add speculative decoding at your actual batch size. If batch size is below ~16, you'll likely see 1.5-2.5× TPOT improvement. Profile acceptance rate — if it's below 0.5, the overhead may not be worth it for your workload.

7. If cost is still too high after the above: Consider whether a smaller model at higher quality is achievable via fine-tuning, whether model routing can send cheaper requests to a distilled model, or whether serving on a multi-tenant GPU cloud gives better amortization than dedicated instances.

Choosing the right serving engine is the other half of this equation — the serving engine comparison article covers vLLM, SGLang, TensorRT-LLM, and Ollama with their actual benchmark numbers and decision criteria. The infrastructure side — Kubernetes deployment, autoscaling, and serverless patterns — is in deploying LLM workloads.

The general pattern: algorithmic improvements (batching, caching, speculative decoding) come before hardware upgrades because they change the cost-per-token equation on existing hardware. Hardware upgrades are amplifiers, not substitutes.

sequenceDiagram
    participant ENG as Engineer
    participant SYS as Serving System
    participant GPU as H100 GPU

    ENG->>SYS: Set max_batch_tokens, chunked prefill enabled
    SYS->>GPU: Prefill chunk 1 (512 tokens) for request A
    GPU-->>SYS: KV cache for chunk 1
    SYS->>GPU: Decode step for requests B, C, D (parallel)
    GPU-->>SYS: 3 output tokens (B, C, D each advance 1 step)
    SYS->>GPU: Prefill chunk 2 for request A + decode for B, C, D
    GPU-->>SYS: KV cache chunk 2 + 3 more output tokens
    Note over SYS,GPU: Request B finishes — new request E enters the batch immediately
    SYS->>GPU: Decode for C, D, E + prefill chunk 3 for A
    GPU-->>SYS: 3 output tokens + final KV cache

The GPU is never idle waiting for a long prefill to clear, and no completed request sits in a finished slot while the batch fills again. That's continuous batching plus chunked prefill together, and it's the baseline that every other optimization builds on.