Choosing a serving engine: vLLM, SGLang, TensorRT-LLM, and Ollama in 2026

In November 2024 a team running a customer-support bot on Text Generation Inference did a load test at 100 concurrent users and saw ~350 tokens/second. They assumed that was the ceiling for their H100 pair. It wasn't — it was the ceiling for TGI. Migrating to vLLM on the same hardware three weeks later gave them ~900 tokens/second. That's a 2.5× throughput gain, same model, same GPU, same cloud bill — the only thing that changed was the software between the GPU and the internet.

This is now a common story. The serving engine is not a commodity layer. It determines your memory fragmentation, your batching efficiency, your tail latency, and ultimately your cost per token. The four engines worth understanding in mid-2026 each make different bets.

What a serving engine actually does

A serving engine is responsible for seven things — and most of the performance differences between engines come down to how they handle steps 3 and 5. When a request arrives at an LLM serving endpoint, the engine must:

1. Accept the incoming prompt and tokenize it
2. Schedule the request into a running batch (or start a new batch)
3. Allocate GPU memory for the KV cache this request will produce
4. Run the prefill forward pass (compute attention over the full prompt, producing the first KV entries)
5. Run autoregressive decode steps, one token per step, reading the full model weights and all KV cache entries on each step
6. Free the KV cache memory when the sequence finishes
7. Stream tokens back to the caller

The subtle insight — covered in depth in Prefill vs Decode: The Two Phases of Every LLM Request — is that steps 4 and 5 are completely different computational regimes. Prefill is compute-bound (large parallel matmuls over all prompt tokens at once). Decode is memory-bandwidth-bound (loading the full model weights plus KV cache on every single step). The serving engine's job is to orchestrate these two regimes efficiently across many concurrent requests.

Poor orchestration wastes GPU time in two main ways: memory fragmentation (allocating KV cache in chunks that leave unusable gaps) and batch underutilization (waiting for some requests to finish before admitting new ones). Every engine in this article has a different approach to solving both.

vLLM: the production default

vLLM's core innovation, PagedAttention, treats the KV cache the way an operating system treats virtual memory. Instead of allocating a contiguous block per request upfront — which fragments GPU RAM badly when request lengths vary — it divides the KV cache into fixed-size pages (typically 16 tokens per page) and allocates pages on demand. When a request finishes, its pages are returned to the pool immediately. The result is near-zero internal fragmentation: vLLM routinely achieves 90%+ KV cache utilization where a naively-allocated system might hit 40%.

sequenceDiagram
    participant REQ as Incoming Request
    participant SCHED as vLLM Scheduler
    participant PAGE as Page Pool (GPU HBM)
    participant EXEC as CUDA Execution

    REQ->>SCHED: prompt (variable length)
    SCHED->>PAGE: allocate N pages (prefill estimate)
    PAGE-->>SCHED: page indices
    SCHED->>EXEC: prefill forward pass (all prompt tokens)
    EXEC-->>PAGE: write KV entries to allocated pages
    loop decode steps
        SCHED->>PAGE: allocate 1 page (if needed)
        SCHED->>EXEC: decode step (one new token)
        EXEC-->>REQ: stream token
    end
    REQ-->>PAGE: free all pages on completion
    Note over PAGE: pages returned to pool immediately

The V1 engine (2025) added chunked prefill as the default behavior. Long prefills — say, a 32K-token RAG prompt — are sliced into fixed chunks (default 512 tokens) that interleave with decode steps from other requests. Without chunked prefill, a single long prefill can monopolize the GPU for hundreds of milliseconds, stalling every other request's decode. With it, p95 TTFT drops ~68% on 32K-token inputs at the cost of ~5% throughput overhead. That is a very good trade for latency-sensitive workloads.

V1 also added disaggregated prefill via NixlConnector, allowing prefill to run on a dedicated GPU pool while decode runs on a separate pool. This is the technique behind the 75% throughput gain measured on H100+H200 pairs. See Prefill vs Decode for the mechanics; the short version is that routing the two phases to hardware matched to each phase's bottleneck (compute for prefill, bandwidth for decode) releases throughput that colocation can't reach.

What vLLM does well:

• Broadest model support: any HuggingFace architecture, NVIDIA/AMD/Intel/TPU
• All major quantization formats: AWQ, GPTQ, FP8, GGUF
• OpenAI-compatible REST API out of the box
• Multi-LoRA serving — hundreds of fine-tuned adapters from one base weight copy in memory
• The most active open-source contributor base, which matters for quickly supporting new model architectures (Llama 4, Qwen 3, DeepSeek V3 all had vLLM support within days of release)

Where vLLM falls short:

• Prefix caching (added in v0.4+) is automatic but less aggressive than SGLang's RadixAttention — it caches exact-match prefixes but doesn't build a trie across the full request history
• Default configurations are generic and need tuning for specific workload shapes

SGLang: the prefix-sharing specialist

SGLang's architectural bet is that real production workloads have massive prefix redundancy that generic paging misses. Every chatbot sends the same 2K-token system prompt thousands of times per hour. Every RAG pipeline has a 500-token retrieval template. Every agent framework preambles with a 1,500-token tool spec. The naive approach computes KV entries for that prefix once per request. SGLang's RadixAttention builds a radix tree (a compressed trie) over all active KV cache prefixes, so a shared prefix is computed exactly once and reused by every request that shares it.

flowchart LR
    subgraph "Naive (vLLM v0.3 / TGI)"
        R1A["Request 1\n[SYS_PROMPT][user turn 1]"] -->|compute KV| KV1["KV cache\nfull sequence"]
        R2A["Request 2\n[SYS_PROMPT][user turn 2]"] -->|compute KV| KV2["KV cache\nfull sequence"]
        R3A["Request 3\n[SYS_PROMPT][user turn 3]"] -->|compute KV| KV3["KV cache\nfull sequence"]
    end
    subgraph "SGLang RadixAttention"
        SYS["[SYS_PROMPT] KV\n(computed once)"] -->|shared| Q1["[user turn 1] KV"]
        SYS -->|shared| Q2["[user turn 2] KV"]
        SYS -->|shared| Q3["[user turn 3] KV"]
    end

The effect is dramatic on prefix-heavy workloads. On chatbot and RAG traffic shapes, SGLang v0.5.8 benchmarks show ~29% higher throughput than vLLM and meaningfully lower TTFT tails, because shared-prefix requests skip the prefill phase entirely for the shared portion. On random-prefix traffic (every request has a unique, unshareable prompt), SGLang and vLLM perform roughly equivalently — RadixAttention just degenerates to standard paging.

SGLang also ships first-class support for EAGLE-2 speculative decoding — a launch flag away for models with published EAGLE head weights. EAGLE-2 runs a small autoregressive draft head on the target model's own hidden features (no separate draft model), proposes a dynamic tree of draft tokens per step, and has the target verify them in one pass. The speedup depends on acceptance rate but typically yields 1.5–2.5× decode throughput at low batch sizes. See Speculative Decoding for the full mechanics.

Easy to forget amid the throughput numbers: the "SGL" in SGLang stands for Structured Generation Language, the project's original frontend for grammar-constrained decoding. If your workload is agents or tool use emitting JSON on every step, SGLang's native constrained generation masks invalid tokens during decode itself — no retry loops, no post-hoc validation failures — and composes with RadixAttention since the schema prompt is itself a shared prefix. vLLM covers the same ground through its xgrammar and Outlines backends; TensorRT-LLM's support is thinner.

On DeepSeek-R1 and similar reasoning models with long chain-of-thought outputs, SGLang has consistently led independent benchmarks through early 2026. The combination of long shared prefixes (reasoning model system prompts are verbose), EAGLE-2 decode acceleration, and FP8 native support on H100 makes it a strong match for that workload class.

The tradeoff: SGLang's non-NVIDIA support lags vLLM. If you have AMD MI300X or Intel Gaudi hardware, vLLM has more mature support. SGLang has been part of the PyTorch ecosystem since March 2025, which accelerates its hardware coverage — but as of mid-2026, vLLM is the safer choice for non-NVIDIA hardware.

TensorRT-LLM: maximum NVIDIA throughput

TensorRT-LLM's approach is fundamentally different from the other two: instead of running standard PyTorch operations and relying on runtime optimization, it compiles your model into CUDA kernels specifically optimized for your exact GPU architecture. The attention computation, the MLP layers, and the decode loop are all fused and tuned for H100 SXM vs H100 PCIe vs A100 vs whatever you specify at build time.

The result is the highest peak throughput available on NVIDIA hardware — 20–40% above vLLM on typical large-model workloads — but you pay for it in operational complexity:

• Building a model takes 30–90 minutes (large models, sometimes longer)
• The build is version-pinned: a specific CUDA version, TensorRT version, and model checkpoint
• Rolling back requires keeping old build artifacts around or rebuilding
• Supported architectures are explicitly listed; new models may lag by days or weeks

TensorRT-LLM operational overhead (illustrative for a 70B model team)
──────────────────────────────────────────────────────────────────────
Build time per model version:          ~60 min on 8× A100
Storage for build artifacts:           ~150 GB per build
CUDA/TRT version compatibility:        1 TRT major version ≈ 3-6 month support window
Model update cadence:                  Any fine-tune checkpoint → new build
Canary deploy workflow:                Maintain 2 build artifact sets simultaneously

Compare: vLLM update workflow:         docker pull + restart (minutes)

This overhead is genuinely worth it when throughput is the primary constraint and you have dedicated MLOps capacity to maintain the build pipeline. NVIDIA's inference team uses TensorRT-LLM internally and publishes the performance data to back it up. But most product teams hit vLLM or SGLang throughput ceilings later than they expect, and the build pipeline complexity arrives immediately.

A common pattern: start on vLLM, measure throughput at production load, and only invest in TensorRT-LLM when benchmarks show a measurable gap that would change your economics. Don't build the pipeline speculatively.

Ollama: local dev, not production

Ollama makes running a local LLM trivially easy: ollama run llama3.1 downloads the GGUF model and starts a server. For a developer who wants to test prompts against a local model without a cloud API, it is excellent.

The architectural problem for production is the scheduler, not the absence of concurrency. Ollama can run a handful of requests in parallel (configurable via OLLAMA_NUM_PARALLEL, on top of llama.cpp's batching), but the parallelism is a small fixed cap with no continuous-batching scheduler and no PagedAttention-class KV memory management. At 10 concurrent users you are already past the cap and requests queue. At 100, latency grows linearly with queue depth. The Continuous Batching and Scheduling article covers why this matters — the short version is that a model with no batching leaves the GPU mostly idle between token generations, achieving a fraction of its theoretical throughput.

Ollama is also GGUF-only. For a production server with AWQ or FP8 weights, Ollama can't load the model at all. And it lacks the monitoring, scheduling, and multi-LoRA machinery that production deployments need.

Use Ollama on your laptop. Use vLLM or SGLang in production. These are different tools for different problems.

What the benchmarks actually show

The instinct after reading benchmarks is to pick the fastest-looking number. The trap is that most published benchmarks test throughput at a specific batch size with synthetic uniform-length prompts, which doesn't reflect real traffic.

Real workloads have:

• Heavy-tailed length distributions (most prompts short, some extremely long)
• Mixed prefill/decode ratios (chatbots are decode-heavy, RAG is prefill-heavy)
• Prefix sharing that varies from 0% (random prompts) to 80%+ (templated agent workflows)
• Latency SLAs that require p95 TTFT and TPOT, not just peak tok/s

The right benchmark methodology is to run your actual traffic distribution (or a sampled replay) against candidate engines at realistic concurrency levels (100–200 simultaneous users), measuring p50/p95 TTFT and TPOT, not just throughput. A synthetic benchmark that shows Engine A is 20% faster than Engine B at batch=32 will tell you almost nothing if your real traffic has 5% prefix sharing and p95 requests are 10x the average length.

flowchart TD
    BENCH[Start benchmarking] --> TRAFFIC{Capture real\ntraffic sample}
    TRAFFIC -->|Yes| REPLAY[Replay at\n100-200 concurrent users]
    TRAFFIC -->|No available| SYNTH[Synthetic traffic:\nmatch real p50/p99\nlength distribution]
    REPLAY --> METRICS[Measure: p50/p95 TTFT\np50/p95 TPOT\nthroughput tok/s\nGPU utilization %]
    SYNTH --> METRICS
    METRICS --> COMPARE[Compare engines\non YOUR workload]
    COMPARE --> DECISION[Pick engine,\ntune chunk size +\nbatch tokens for winner]
    style METRICS fill:#00e5ff,color:#0a0a0f

KV cache memory: the shared constraint

All engines compete for the same GPU HBM, and the KV cache is the dominant consumer after model weights at any real concurrency. The formula — covered in detail in The KV Cache: GPU Memory Math Every Serving Engineer Must Know — is:

KV bytes per token = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element

Llama 3.1 70B (BF16, GQA with 8 KV heads, 80 layers, 128 head_dim):
= 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes ≈ 0.32 MB per token

At 100 concurrent users × 8K context average:
= 100 × 8,192 × 0.32 MB ≈ 262 GB of KV cache alone

Two H100 SXM (80 GB each) = 160 GB total HBM
Model weights at BF16: ~140 GB
→ Only 20 GB left for KV cache — supports ~62 concurrent 1K-context requests before eviction

This math determines why quantization choices and memory management strategies matter so much. FP8 weights on H100 cut model footprint to ~70 GB, freeing ~90 GB for KV cache — roughly 4× the concurrent capacity. SGLang's RadixAttention reduces effective KV memory pressure further by not storing redundant prefix copies. These are complementary wins, not alternatives.

For quantization format details, the relevant decision for engine selection is: FP8 is the best choice on H100/H200/B200 (hardware-native, near-BF16 quality), AWQ 4-bit is the right choice for memory-constrained scenarios or older hardware, and GGUF is the only option for CPU-hybrid serving (which means Ollama on a consumer machine).

What breaks

Defaults are wrong for your workload

vLLM and SGLang both ship with generic defaults that work reasonably well across workload types but are rarely optimal for any specific workload. The chunked prefill chunk size (default 512 tokens), the maximum batch tokens limit, and the KV cache block size all have workload-specific optimal values. A chatbot with short prompts and long outputs benefits from smaller chunk sizes and more aggressive batching. A RAG pipeline with 8K-token prompts and 200-token outputs is the opposite. Deploying with defaults and calling the performance numbers "production benchmarks" is a common mistake.

Benchmarking at wrong concurrency

The other frequent mistake: measuring throughput at batch=1 or batch=8 and reporting it as the system's capability. Real p95 latency at 100+ concurrent users is often 3-5× worse than the single-request latency, because queuing time dominates. Any SLA discussion should be grounded in p95 numbers at realistic concurrency, with real length distributions.

Over-quantizing the wrong layer

If you use speculative decoding (recommended for latency-bound workloads), the draft model's quantization matters. An INT4-quantized draft model can have low enough acceptance rates that the speculative decoding overhead makes total latency worse than no speculation at all. Monitor acceptance rate per request and set a floor. INT8 or FP8 draft is usually the right balance.

TGI migration delay

If your team is still running TGI as of mid-2026, this is the highest-impact serving change available to you. TGI entered maintenance mode in December 2025, and the throughput gap versus vLLM at 100-200 concurrent users ranges from 3× to 24× depending on workload. The migration is a Docker image swap and API configuration change — the OpenAI-compatible endpoint shapes are largely identical. The performance gain is not theoretical.

VRAM accounting for the full picture

A 70B model in BF16 needs 140 GB for weights. That leaves 20 GB across two H100s for KV cache. At 8K context average, an 8K-token KV footprint is ~2.6 GB per user — you support roughly 8 concurrent users (or ~60 at 1K context, per the worked block above) before eviction kicks in and throughput collapses. Teams that provision based on "does the model fit?" without accounting for KV cache at target concurrency routinely hit OOM crashes or severe memory pressure in production. The numbers above are straightforward to compute; do them before sizing your cluster, not after the first prod incident.

The decision in practice

The decision tree in the summary block gives the answer in four questions. Here is the same logic with more texture.

Start with vLLM if you need to ship something and don't have specific benchmarks showing it's insufficient. It handles the 80% case, has the most documentation and community support, and works across hardware vendors. The V1 engine is mature enough for production as of mid-2026, and chunked prefill is on by default.

Move to SGLang when you have measurement showing prefix sharing is substantial (>30% of your KV cache would be shared across concurrent requests) or when you're running reasoning models that generate long chain-of-thought sequences. The 29% throughput improvement on those workloads is real and consistent across independent benchmarks. SGLang's EAGLE-2 speculative decoding support is also a meaningful latency win for low-batch-size scenarios.

Evaluate TensorRT-LLM only after you've measured that vLLM or SGLang is the actual bottleneck at your target throughput, and you have the MLOps capacity to own a compiled build pipeline. The performance ceiling it exposes is real, but so is the operational cost. Most teams that invest in TensorRT-LLM find the break-even point arrives later than expected because vLLM V1 and SGLang caught up significantly in 2025.

Keep Ollama on laptops. It's genuinely good for what it's designed for. Running it as a production API is like using SQLite for a high-traffic web backend — fine at zero scale, bad at any real load.

One final note on the competitive dynamics: SGLang and vLLM are converging. vLLM's prefix caching has improved in every release through 2025-2026, and SGLang now benefits from being in the PyTorch ecosystem with faster hardware coverage expansion. The gap that exists today may look different in six months. The mechanism that explains the gap — RadixAttention vs standard paging — will help you re-evaluate as the benchmarks update, rather than taking any snapshot comparison as settled truth.

For the next layer on throughput vs latency GPU economics and deploying these engines on GPUs and Kubernetes, those sibling articles pick up where this one stops.