Continuous Batching and Scheduling: Why Static Batching Wastes 70% of Your GPU

A team at a mid-size AI startup spent three months optimizing their serving setup — quantizing weights, tweaking temperature, tuning prompt length. Their cost per query dropped 12%. Then they switched from static to continuous batching and it dropped another 61% in a single afternoon. The model did not change. The hardware did not change. The only thing that changed was when the scheduler decided to admit new requests. That single scheduling rule is why this article exists.

Why static batching fails under real traffic

Every GPU forward pass processes a batch of sequences in parallel. Batching is how you amortize the fixed cost of loading model weights from HBM into registers — running one request at a time means loading 70B parameters of weights on every single decode step, most of which are immediately discarded. Pack 32 requests into one batch and you load those weights once and do useful work 32 times. Throughput scales almost linearly with batch size, up to the memory bandwidth ceiling.

Static batching does this correctly, but with a fatal structural problem: it treats the batch as atomic.

Here is what static batching does:

sequenceDiagram
    participant Q as Request Queue
    participant S as Scheduler
    participant G as GPU

    Q->>S: Req A (output: 200 tok), Req B (10), Req C (60), Req D (35)
    S->>G: Start batch {A, B, C, D}
    Note over G: Step 10 — B finishes
    Note over G: Slot B idles (waiting for batch to complete)
    Note over G: Step 35 — D finishes
    Note over G: Slot D idles
    Note over G: Step 60 — C finishes
    Note over G: Slot C idles
    Note over G: Step 200 — A finishes
    G->>S: Batch done
    S->>Q: Admit next batch

Sequence B finishes at step 10. But static batching does not release that slot until step 200, when the last sequence finally ends. For 190 steps, one-quarter of the batch is burning electricity doing nothing.

Now run real production traffic through that model. Output lengths follow a heavy-tailed distribution — most requests are short, a few are very long. In practice, a typical production workload with a max output length of 2,048 tokens might see a median output of 80 tokens and a p99 of 1,500. When request A draws the short straw and outputs 1,500 tokens, every other sequence in that batch sits idle for 1,400 extra steps. GPU utilization collapses.

The prefill vs decode phases article covers why decode is memory-bandwidth-bound. What matters here is that each decode step is fast — often 20-40 ms for a full batch on an A100 — and the GPU is available between steps. Static batching throws that availability away.

Continuous batching: the fix

Continuous batching (also called in-flight batching or iteration-level batching) changes one rule: a slot does not have to wait for the rest of the batch.

The moment any sequence emits its <EOS> token or hits its max_tokens limit, the scheduler treats that slot as free at the next iteration boundary and immediately fills it with the next queued request. The GPU never stalls.

sequenceDiagram
    participant Q as Request Queue
    participant S as Scheduler
    participant G as GPU

    Q->>S: Req A (200 tok), Req B (10), Req C (60), Req D (35)
    S->>G: Start iteration {A, B, C, D}
    Note over G: Step 10 — B finishes, slot freed
    Q->>S: Req E waiting
    S->>G: Iteration {A, E, C, D} — E replaces B
    Note over G: Step 35 — D finishes, slot freed
    Q->>S: Req F waiting
    S->>G: Iteration {A, E, C, F}
    Note over G: Step 60 — C finishes, slot freed
    Note over G: ...GPU stays full until queue drains

The GPU is always doing real work. Every step, every slot is occupied by a request that is actively generating tokens.

The throughput numbers from early deployments were dramatic enough that the industry moved fast. The Orca paper (OSDI 2022) introduced the term "iteration-level scheduling" and reported up to ~37x higher throughput than NVIDIA's FasterTransformer, which processed batches statically. When vLLM shipped PagedAttention in mid-2023, its launch benchmark on LLaMA-7B and 13B became the number everyone quotes: up to 24x higher throughput than naive HuggingFace Transformers batching, and roughly 3.5x over TGI (HuggingFace Text Generation Inference) — which already did continuous batching but managed KV cache memory far less efficiently. The gap over static batching widens with concurrency, because a static batcher saturates and starts queueing while the continuous scheduler keeps filling slots.

{ "type": "batching", "mode": "static", "title": "Static vs continuous batching: GPU utilization gap" }

Why continuous batching requires PagedAttention

There is a coupling between the scheduler and KV cache management that is easy to miss.

Under static batching, you can pre-allocate KV cache memory for each sequence at batch formation time. You know the maximum output length (or pad to it), so you reserve a contiguous block of GPU HBM for each request. Simple.

Under continuous batching, you do not know when a request will arrive or how long its output will be. Pre-allocating max_output_length for every sequence wastes enormous amounts of HBM — and since KV cache is the dominant memory consumer in a serving deployment, this matters a lot.

PagedAttention (the core vLLM innovation) solves this by treating KV cache like virtual memory pages. Instead of one contiguous block per sequence, the KV cache is divided into fixed-size pages (blocks), and each sequence gets pages allocated dynamically as it generates more tokens. When a sequence finishes, its pages are immediately reclaimed and available for the next request.

flowchart TD
    subgraph PrePage["Without PagedAttention"]
        R1["Req A: reserves 2,048 tokens of KV"]
        R2["Req B: reserves 2,048 tokens of KV"]
        R3["Req C: reserves 2,048 tokens of KV"]
        R1 -.->|"actual output: 60 tokens\n1,988 blocks wasted"| WASTE["HBM fragmentation"]
    end
    subgraph WithPage["With PagedAttention"]
        P1["Req A: 3 pages allocated, grows on demand"]
        P2["Req B: 1 page → done → pages reclaimed"]
        P3["Req C: 4 pages → done → pages reclaimed"]
        P1 --> POOL["Page pool — zero fragmentation"]
        P2 --> POOL
        P3 --> POOL
    end

PagedAttention is what makes continuous batching practical at scale. Without dynamic page allocation, the KV cache memory pressure of admitting requests continuously would cause OOM crashes or force such conservative batch sizes that the throughput gains disappear. With it, the engine maintains high batch occupancy without reservation waste.

The scheduling layer: FCFS, preemptive, priority

Continuous batching decides when to admit a request. The scheduler decides which request to admit and what happens when the KV cache fills up.

FCFS (first-come, first-served) is the simplest policy. Requests join the running batch in arrival order. When the KV cache is full, new arrivals queue — and if the pool fills mid-flight, the scheduler still preempts the most recently admitted sequences rather than crashing. Preemption is a memory-pressure mechanism, not a separate opt-in policy. FCFS is vLLM's default and works well for homogeneous traffic where fairness matters.

Preemptive scheduling goes further: if the KV cache is under pressure, the scheduler can pause a running sequence — either swapping its KV pages to CPU RAM or flagging it for recomputation — to make room for a higher-priority or newer request. Swapping is slower (PCIe bandwidth is 32-64 GB/s, versus HBM's 3+ TB/s), and recomputation costs prefill FLOPS. So preemption has a throughput cost. But it prevents a single long-running sequence from monopolizing KV memory and starving the queue, which improves tail latency under bursty traffic.

Priority scheduling assigns tiers to requests — interactive user sessions vs background batch summarization jobs, for example. Interactive requests get a slot immediately; batch jobs fill remaining capacity. This lets you mix workloads on one GPU pool without a dedicated instance per workload type.

The key insight: none of these are about correctness. They are about where the latency pain lands. Pick FCFS for simplicity, preemptive if your p95 TTFT SLA is strict, priority if your traffic is genuinely heterogeneous.

Chunked prefill: the missing piece for long prompts

Continuous batching solved the output-length variance problem. But it left a related problem untouched: prefill blocks the batch.

When a request with a 32K-token prompt arrives, the prefill step — processing all 32,000 tokens to build the initial KV cache — runs as a single, undivided GPU operation that can take hundreds of milliseconds. During that time, all the decode sequences in the batch are stalled, waiting. For short prompts this is negligible. For 8K+ token prompts (RAG with large retrieved contexts, long documents, complex agents with full conversation history), it becomes the dominant contributor to TTFT variance for everyone sharing that GPU.

Chunked prefill slices the prefill into fixed-size chunks — say 512 tokens — and interleaves them with decode steps from other sequences. The prompt is still fully processed; it just yields the GPU between chunks.

sequenceDiagram
    participant D as Decode Sequences (A, B, C)
    participant P as Long Prompt (32K tokens)
    participant G as GPU

    Note over G: Without chunked prefill
    G->>P: Full 32K prefill (300ms — decode blocked)
    G->>D: Resume decode

    Note over G: With chunked prefill (512 tok chunks)
    G->>P: Chunk 1 (512 tok, ~5ms)
    G->>D: Decode step (A, B, C generate one token each)
    G->>P: Chunk 2 (512 tok)
    G->>D: Decode step
    Note over D: Sequences keep streaming — TTFT variance shrinks

The Sarathi-Serve paper (OSDI 2024) measured what this buys: eliminating prefill stalls cuts tail time-between-tokens for the already-decoding sequences by an order of magnitude on long-prompt workloads, and raises how much traffic a GPU can serve within a latency SLO. Requests queued behind a long prefill see their TTFT improve too — the chunked request itself pays a small first-token delay in exchange. vLLM V1 (shipped 2025) enabled chunked prefill as the default configuration. If you are running an older vLLM version with --disable-chunked-prefill or have not upgraded, this is low-hanging fruit.

Numbers: what the scheduler is actually doing to your cost

Let's make this concrete with a real cost calculation.

Setup: Llama 3.1 70B on 2× H100 80GB SXM (tensor parallel)
GPU cost: $8/hr per H100 → $16/hr total
Measured throughput:
  Static batching (naive per-batch):  ~800 tok/s at batch=32
  Continuous batching (vLLM):         ~3,200 tok/s at batch=32+

Cost per million output tokens:
  Static:     ($16/hr) / (800 tok/s × 3600 s/hr) × 1,000,000 = $5.56 / M tok
  Continuous: ($16/hr) / (3,200 tok/s × 3,600 s/hr) × 1,000,000 = $1.39 / M tok

Savings: 75% reduction in cost per token, same hardware, same model.

At 50B output tokens/month (fleet scale — roughly two dozen GPU pairs on static batching):
  Static:     $278,000 / month
  Continuous: $69,500 / month
  Delta:      $208,500 / month saved

These numbers are illustrative — real throughput varies with traffic shape, model architecture, and hardware — but the direction is representative. The throughput ratio between static and continuous batching grows with concurrency; this estimate uses a moderate concurrency level. At 200+ concurrent requests, the gap widens further.

The throughput vs latency economics article goes deeper on how to model GPU cost against SLA requirements. The short version: continuous batching shifts cost per token dramatically, but you still need to tune max_batch_tokens to hit your TPOT SLA at target concurrency.

What breaks

KV cache OOM under sudden traffic spikes. Continuous batching admits requests aggressively. If a burst of long-prompt requests arrives simultaneously, the KV page pool can fill faster than the scheduler can preempt. vLLM handles this with configurable gpu_memory_utilization (default 0.9, not 1.0) and preemption policies, but if you do not set appropriate max_model_len and max_num_seqs, a traffic spike can crash the server with OOM. Monitor KV cache utilization as a first-class metric alongside GPU utilization.

Head-of-line blocking during prefill. Without chunked prefill, a single request with a very long prompt (32K+ tokens) stalls decode for every other sequence in the batch. At p99, this adds hundreds of milliseconds to TTFT for concurrent users. Enable chunked prefill; tune chunk size to your typical prompt length distribution.

TPOT degradation at large batch sizes. Continuous batching maximizes batch size, which is great for throughput but increases TPOT because the GPU is splitting compute across more sequences. For interactive applications with a strict TPOT SLA (say, <40 ms/token for a perceptible streaming experience), you need to cap max_batch_tokens even if it means lower throughput. This is not a bug — it is the fundamental throughput/latency trade-off, and you are now in control of where it sits.

Benchmark/production mismatch. If you benchmark continuous batching with synthetic uniform-length prompts, you will underestimate throughput gains and overestimate TPOT in production. Real traffic has heavy-tailed length distributions. Always benchmark with a traffic replay or a distribution that matches your p50/p95/p99 input and output lengths. A benchmark with average-length prompts overestimates throughput and hides p99 TTFT surprises.

Multi-LoRA scheduling complexity. If you are serving multiple fine-tuned LoRA adapters from a single base model (increasingly common for per-customer personalization), the scheduler must now track which adapter each request needs and ensure the right adapter weights are loaded before that slot executes. vLLM V1 and SGLang both support multi-LoRA serving, but the scheduler overhead is real and adapter switching adds latency. Plan for this if your use case involves more than a handful of adapters.

TGI in production (the maintenance-mode trap). Text Generation Inference entered maintenance mode in December 2025. To be clear about what TGI is and is not: it has done continuous batching since mid-2023 — the famous vLLM-vs-TGI gap came from PagedAttention's memory management, not from TGI lacking iteration-level scheduling. But it fell behind on KV cache management and kernel optimization, and teams still running it are on unsupported software giving up a small-multiple (~2-4x-class) throughput gap versus vLLM or SGLang. Migration is a weekend project for most deployments; the API surfaces are compatible enough that the change is largely configuration.

The current state of the scheduler (2025-2026)

Three major engines implement continuous batching with different emphases, covered in detail in the serving engine comparison:

vLLM V1 (2025) restructured the scheduler around async execution — the Python scheduler and GPU execution are now decoupled, reducing scheduling overhead by ~20% at high batch sizes. Disaggregated prefill/decode via the NIXL connector lets you run prefill on dedicated H100 instances and decode on H200 instances, with a ~75% measured throughput gain on H100+H200 pairs at the cost of inter-GPU network traffic. Chunked prefill is default.

SGLang's core differentiator since its 2024 debut is RadixAttention, which extends the KV cache to share pages across different requests that share a common prefix (a long system prompt, for instance). For workloads with large shared prefixes — RAG with a fixed retrieval template, agent frameworks with a static tool list — this cuts KV memory pressure by the fraction of the context that is shared, and improves effective throughput by 29% over vLLM on those workloads.

TensorRT-LLM achieves the highest peak NVIDIA throughput via compiled CUDA kernels, but the per-model build pipeline adds rollback complexity and deployment friction. It is only worth the operational overhead when benchmarks show vLLM or SGLang genuinely cannot meet your SLA on the same hardware.

The scheduling abstraction itself is stable; the implementation is still being optimized rapidly. Whatever numbers exist in vendor benchmarks today will be out of date in six months — anchor to the mechanism, not the specific figures.

What preemption actually costs

Preemption deserves more detail because it is frequently misunderstood.

When the scheduler preempts a running sequence, it has two options:

Swap to CPU. The sequence's KV pages are moved from GPU HBM to CPU DRAM over the PCIe bus (32-64 GB/s peak, usually lower in practice). For a sequence with 4K tokens of KV cache at BF16, that is roughly 4,000 × 0.32 MB (using Llama 3.1 70B's ~0.32 MB/token KV footprint) = ~1.28 GB of data to move. At 32 GB/s, that takes ~40 ms. When the sequence is resumed, the same transfer happens in reverse. Preemption adds ~80 ms per event to that sequence's latency — acceptable if the sequence was about to OOM the pool but not free.

Recompute. Drop the KV pages entirely and re-run the prefill when the sequence resumes. Costs one full prefill pass (compute-bound, fast) but wastes the FLOPS spent on earlier decode steps. This is cheaper than swap for short sequences with short prefills, expensive for long ones.

vLLM's default preemption mode is recompute; swap is a configuration option, historically used where recomputation is impossible (beam-search sequence groups), not something the engine auto-selects by sequence length. The practical takeaway: if your workload is preempting frequently (visible in vLLM's /metrics as preempted_requests_total climbing), you are either under-provisioned on KV cache memory or your max_num_seqs is too high. Reduce max_num_seqs or add GPU HBM before tuning preemption strategy.

The scheduler in practice: tuning levers

For most teams, the defaults in vLLM are close enough to start. The levers that actually move the needle:

• max_num_seqs: maximum number of sequences in the running batch simultaneously. Increasing this improves throughput at high concurrency but increases per-request TPOT and KV cache pressure. Tune based on your TPOT SLA.
• max_model_len: maximum total sequence length (input + output). Caps KV allocation per sequence. Set to the 99th percentile of your actual traffic, not the model's theoretical maximum.
• gpu_memory_utilization: fraction of GPU HBM to allocate to the KV page pool. Default 0.90. Higher values increase concurrency; values above 0.95 risk OOM under burst traffic.
• enable_chunked_prefill: enabled by default in vLLM V1. If on an older version, set explicitly. The chunk size (max_num_batched_tokens) should be tuned to your typical prefill length — too small and you add scheduler overhead; too large and long prompts still block briefly.

For SGLang, the equivalent parameters are --max-running-requests, --mem-fraction-static, and --chunked-prefill-size. The concepts are identical; only the flag names differ.

When to use what

Use continuous batching always. There is no production use case where static batching is preferable — the throughput loss under any real traffic shape is too large to justify.

Use chunked prefill if your median input length exceeds ~2K tokens. Below that, the scheduling overhead of chunking may slightly reduce throughput without meaningful TTFT benefit. Above 4K, it is non-negotiable.

Use preemptive scheduling when you have SLA-sensitive interactive traffic coexisting with long-running batch jobs on the same GPU pool. Otherwise FCFS is simpler and has less overhead.

Use RadixAttention (SGLang) if a significant fraction of your traffic shares a large common prefix — long system prompts, fixed RAG templates, shared agent instructions. The KV cache savings can be substantial.

Use disaggregated prefill/decode when you are operating at scale (dozens of GPU instances) and have measured that prefill contention is your binding constraint. Below that scale, the network overhead between prefill and decode instances is not worth the operational complexity.

Continuous batching is the baseline. Everything else is additive optimization on top of it. Get the baseline right first — the serving engine comparison covers which engine to use to get it — and you will have recovered the 70% of GPU capacity that static batching was discarding before you touch anything else.

The math above showed $208,500 per month in savings for a fleet-scale 70B deployment. If you are still on static batching, the same 75% per-token reduction applies to your bill, whatever the scale.