Speculative decoding: 2-3x latency speedup without changing model quality

Your Llama 3.1 70B deployment is running at about 22 tokens per second for a single user — right at the H100's HBM bandwidth ceiling. Your p95 TTFT is fine — prefill is fast — but the decode stream feels sluggish. The user is watching text trickle out. You check GPU utilization and it's at 95%. Adding more GPUs would help throughput but not single-user latency. Quantizing the model is an option but you've already read the tradeoffs article and the quality hit on your reasoning workload is borderline.

Speculative decoding is the other lever. It doesn't change the model, doesn't change the outputs, and on low-to-medium batch workloads routinely delivers 2-3x decode speedup. The math is surprisingly clean once you see it.

Why decode is hard to speed up

Every token in the decode phase requires one full forward pass through all model layers — loading all weights from GPU high-bandwidth memory (HBM) into the matrix multiply units. For a 70B model in BF16 that's ~140 GB of weight data per step. On an H100 SXM with 3.35 TB/s HBM bandwidth, that's roughly 42ms of pure memory-read time per forward pass, which gives a theoretical ceiling of ~24 tokens/s. Actual measured single-request decode on 70B BF16 runs at about 20-25 tok/s.

You can't parallelize across token positions during decode: token t+1 depends on token t, so every step is sequential. Increasing batch size helps amortize the weight-load cost across multiple users, which is why continuous batching is so effective for throughput. But for a single user's request, there's no batch to amortize over. You're paying 140 GB of HBM reads per token no matter what.

Speculative decoding's insight: what if you could get multiple tokens out of that one weight-load? The target model's forward pass already computes logits for every position you feed it. If you feed it 5 candidate positions, you get 5 sets of logits in one pass — only marginally more expensive than one position, since the memory bottleneck dominates.

The trick is generating good candidate tokens cheaply before the expensive target pass.

The algorithm, precisely

Here is what actually happens in one speculative decoding step:

sequenceDiagram
    participant D as Draft model
    participant T as Target model
    participant RS as Rejection sampler

    Note over D: Fast autoregressive pass
    D->>D: generate t₁ with prob q(t₁|context)
    D->>D: generate t₂ with prob q(t₂|context, t₁)
    D->>D: generate t₃, t₄, t₅ ...
    D->>T: send [t₁, t₂, t₃, t₄, t₅] as candidates

    Note over T: One parallel forward pass
    T->>T: compute p(t|context) for each position
    T->>RS: [p₁, p₂, p₃, p₄, p₅]

    Note over RS: Token-by-token acceptance
    RS->>RS: accept t₁ with prob min(1, p₁/q₁)
    RS->>RS: accept t₂ with prob min(1, p₂/q₂)
    RS->>RS: at first rejection: resample from adjusted dist
    RS-->>T: accepted prefix + one target-sampled token

The acceptance criterion min(1, p(token)/q(token)) is the key. If the target would have given this token a higher probability than the draft did — the draft was conservative — you always accept. If the target would have given it lower probability — the draft was overconfident — you accept proportionally. Any rejected token gets replaced with a sample drawn from the corrected distribution (p - q)+, normalized. This is standard rejection sampling from theoretical statistics, and it provably yields samples with distribution p — identical to sampling from the target directly.

What you get out:

• Every accepted draft token costs zero additional target compute.
• The first rejected token still costs one target pass (the verification pass), but you get a corrected sample from it.
• You always advance at least one token per step (the rejection case), and on average many more.

The expected number of tokens accepted per step is:

E[tokens/step] = 1 + α + α² + α³ + ... + αᴷ

Where:
  α = per-token acceptance rate (0 to 1)
  K = number of draft tokens proposed

At α=0.7, K=5: 1 + 0.7 + 0.49 + 0.343 + 0.240 + 0.168 = 2.94 tokens/step
At α=0.8, K=5: 1 + 0.8 + 0.64 + 0.512 + 0.41 + 0.328 = 3.69 tokens/step
At α=0.9, K=5: 1 + 0.9 + 0.81 + 0.729 + 0.656 + 0.590 = 4.69 tokens/step

Acceptance rate α is the single variable that determines whether speculative decoding is worth deploying. Everything else is secondary.

{ "type": "speculative", "title": "Speculative decoding: acceptance rate vs speedup", "acceptance": 0.75 }

What determines acceptance rate

The draft model and target model need to agree. Obvious when stated plainly, but it has sharp practical implications.

Model family matching matters most. Start with the hard constraint: the rejection sampling step compares p/q token by token over the same vocabulary, so classic draft-model speculation requires the draft and target to share an identical tokenizer — a Mistral draft with a Llama target won't run at all, and vLLM will refuse the pairing. In practice that means same-family models. A Llama 3.2 3B draft paired with a Llama 3.1 70B target gives much higher acceptance than an arbitrary same-tokenizer 3B: both were trained on similar data, and the smaller model has learned to approximate the larger one's behavior. Even with the tokenizer matched, a distribution mismatch — a base-model draft under an instruct-tuned target, or a code-tuned draft on chat traffic — can drop acceptance to 0.4-0.5, which is below break-even.

Temperature tanks acceptance rate. At temperature 1.0 with creative prompts, the target's token distribution is flat — many tokens have similar probability. The draft's top-1 prediction has low p/q, so acceptance falls. Lower temperatures sharpen both distributions, so they agree on the mode more often; at temperature 0.0 (greedy) the acceptance criterion collapses to "did the draft pick the target's argmax?", and acceptance equals the pair's top-1 agreement rate — roughly 0.7-0.9 for a well-matched pair, not 1.0. A 3B model's argmax frequently differs from a 70B model's, even in the same family. Most production deployments run temperature 0.6-0.8, which gives acceptance in the 0.65-0.80 range for well-matched pairs.

Long reasoning chains hurt. If you're running a chain-of-thought workload where the model is generating novel reasoning steps, the draft has a harder time predicting what direction the target will take. EAGLE-2 handles this better than separate-draft methods because it conditions on the target's own hidden states rather than the context alone.

Quantizing the draft buys almost nothing. The draft is already a small fraction of step cost — its FP16 forward pass is cheap next to the target's — so an INT4 draft saves little latency while shifting the draft's distribution and giving up some acceptance. Keep the draft in BF16 or FP16 by default. If VRAM genuinely forces the issue (llama.cpp setups run Q4 drafts routinely and often get away with it), measure acceptance before and after rather than assuming it survives.

Self-speculative decoding: EAGLE-2 and Medusa

The production trend in 2025-2026 is moving away from separate draft models and toward self-speculative methods that generate draft tokens from the target model's own internals.

Medusa attaches multiple independent prediction heads to the last hidden layer of the target model. Each head predicts tokens at positions +1, +2, +3, and so on, in parallel. A tree-based verification scheme then evaluates which draft paths the target model would have taken. Medusa requires fine-tuning (to train the extra heads) but achieves acceptance rates above 0.8 on its training distribution.

EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) goes a step further. Instead of independent heads, EAGLE trains a single lightweight transformer that takes the target model's hidden states as input and predicts the next token at each position. By conditioning on the target's actual internal representations rather than just the output token, EAGLE's predictions are better calibrated to the target's behavior. EAGLE-2 added dynamic draft tree construction — the tree of candidate paths is pruned based on confidence, reducing wasted computation on low-probability branches.

EAGLE-2 is now the default speculative decoding method in SGLang. EAGLE-3 (published 2025) pushes the same idea further — it fuses features from multiple layers of the target instead of just the top and trains the head on direct token prediction, posting higher acceptance than EAGLE-2 on the same workloads; support has landed in both SGLang and vLLM. The practical difference from a deployment perspective: the multi-GB separate draft is replaced by a lightweight head shipped as its own small checkpoint (that's the --speculative-draft-model-path in the SGLang command below), cutting the draft memory footprint from ~6 GB to well under 1 GB, with higher acceptance rates (~0.85-0.90 on typical chat workloads).

flowchart TD
    subgraph "Classic speculative decoding"
        CM1["Draft model\n(separate 3B checkpoint)"] -->|proposes K tokens| CT1["Target model\n(70B)"]
        CT1 -->|verifies in one pass| CO1["Output tokens"]
    end

    subgraph "EAGLE-2 self-speculative"
        EM1["Target model\n(70B)"] -->|hidden states| EH1["EAGLE head\n(lightweight transformer)"]
        EH1 -->|draft tree| EM2["Target model\n(same 70B)"]
        EM2 -->|tree verification| EO1["Output tokens"]
    end

    style CM1 fill:#a855f7,color:#fff
    style CT1 fill:#00e5ff,color:#0a0a0f
    style EM1 fill:#00e5ff,color:#0a0a0f
    style EH1 fill:#a855f7,color:#fff
    style EM2 fill:#00e5ff,color:#0a0a0f

There's also a zero-model option worth knowing: n-gram / prompt-lookup speculation. Instead of running any draft, the server proposes candidates by matching the last few generated tokens against the prompt and copying whatever followed the match. Zero extra VRAM, zero training, no tokenizer constraint — and on input-grounded workloads where the output heavily copies the input (RAG answers quoting retrieved passages, summarization, code edits where the model rewrites a function it was just shown), the copied spans get accepted at high rates and deliver most of the speedup for free. vLLM supports it natively as the ngram speculative method. On open-ended chat it does close to nothing, because the output rarely echoes the prompt. If you're VRAM-constrained and your traffic is RAG or code editing, try this before anything else.

How the methods stack up:

Method	Draft source	Extra VRAM	Training required	Typical acceptance	Caveat
Classic draft model	Separate same-tokenizer model	~6 GB (3B draft)	None	0.6-0.8	Requires identical tokenizer; draft passes add ~20% step time
N-gram / prompt lookup	String match against the prompt	~0	None	High on RAG/summarization/code edits; near-zero on open chat	Only wins when output copies input
Medusa	Parallel heads on last hidden layer	<1 GB	Fine-tune heads	0.8+ on training distribution	Typical-acceptance verification relaxes the strict losslessness guarantee
EAGLE-2 / EAGLE-3	Lightweight head on target hidden states	<1 GB	Trained head (published for common models)	0.85-0.90	Lossless; needs a head checkpoint matching your exact target

Worked numbers: what you actually get

Scenario: Single-user chat, Llama 3.1 70B target, BF16, H100 SXM

Baseline (no speculative decoding):
  Tokens/step: 1.0
  Effective decode rate: ~23 tokens/s
    (HBM ceiling: 3,350 GB/s ÷ 140 GB ≈ 23.9 tok/s; actual measured ~20-25 tok/s)
  100-token response time: TTFT + 100/23 ≈ TTFT + 4.3s

Classic speculative decoding (Llama 3.2 3B draft, K=5):
  Acceptance rate α: ~0.72 (illustrative for chat traffic, temp=0.7)
  E[tokens/step] = 1 + 0.72 + 0.52 + 0.37 + 0.27 + 0.19 = 3.07
  Step time: ~42ms target verify + ~9ms for 5 draft passes ≈ 51ms
  Effective rate: 3.07 tokens / 51ms ≈ ~60 tokens/s
  100-token response time: TTFT + 100/60 ≈ TTFT + 1.7s
  Speedup: ~2.6× on paper — which is why measured numbers land at 2-2.5×

EAGLE-2 self-speculative (SGLang default):
  Acceptance rate α: ~0.86 (on same chat traffic)
  E[tokens/step] = 1 + 0.86 + 0.74 + 0.64 + 0.55 + 0.47 = 4.26
  Step time: ~42ms target pass (tree verification widens it slightly)
    + ~1-2ms draft head ≈ 45ms
  Effective rate: 4.26 tokens / 45ms ≈ ~95 tokens/s
  100-token response time: TTFT + 100/95 ≈ TTFT + 1.1s
  Speedup: ~4× over baseline

Draft model memory overhead (classic, TP=4 across 4× H100-80):
  Target: 140 GB sharded → ~35 GB weights per GPU
  Llama 3.2 3B draft in BF16: ~6 GB (replicated per GPU in the simple setup)
  Per GPU: 80 - 35 - 6 ≈ ~39 GB left for KV cache — workable, but the
  draft comes straight out of KV cache headroom

At batch size 64 (high-throughput regime):
  Target GPU is compute-saturated during verification pass
  Running the draft adds ~8ms overhead per step
  Net effect: 5-10% throughput reduction
  → Disable speculative decoding above batch=16 for this setup

Enabling it in practice

Both vLLM and SGLang expose speculative decoding as a server flag. No application code changes needed.

vLLM (classic draft model):

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-config '{"model": "meta-llama/Llama-3.2-3B-Instruct",
                         "num_speculative_tokens": 5}' \
  --tensor-parallel-size 4

The V1 engine consolidated the old --speculative-model / --num-speculative-tokens flags into the single --speculative-config JSON argument — the old flags error out. Flag shapes drift between releases; check the vLLM docs for your version before copy-pasting.

SGLang (EAGLE-2 self-speculative):

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-70B-Instruct \
  --speculative-algorithm EAGLE \
  --speculative-draft-model-path lmzheng/sglang-EAGLE-llama3.1-instruct-70B \
  --speculative-num-steps 5 \
  --speculative-eagle-topk 4 \
  --tp 4

Both expose the current acceptance rate as a metric. In vLLM it surfaces as speculative_num_accepted_tokens in the Prometheus metrics. In SGLang it appears in the /metrics endpoint. Track this number — a sustained drop below 0.55 is a hard signal that the draft is mismatched to your current traffic pattern, and you're potentially running slower than baseline.

For programmatic use, nothing changes on the client side. The same OpenAI-compatible API works; speculative decoding is entirely internal to the server:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="token")

# identical API — speculative decoding is transparent to the caller
stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-70B-Instruct",
    messages=[{"role": "user", "content": "Explain gradient descent."}],
    stream=True,
    max_tokens=500,
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

What breaks

Draft-target mismatch at traffic shift. Your acceptance rate during load testing with chat traffic was 0.78. You roll out speculative decoding. Three weeks later, your engineering team starts using the endpoint for code generation. Acceptance rate drops to 0.51. Speculative decoding is now running slower than baseline, and the metric tells you — if you're watching it. Set an alert on acceptance rate and have a feature flag to disable speculative decoding per-request-type or globally.

Wrong batch size assumption. You benchmarked at batch=1 (single concurrent user). Production has 50-100 concurrent users. At batch=50, the target GPU is already fully utilized during the verification pass. The draft model adds latency to each step without proportional acceptance gain. Throughput drops 10-15%. The fix is to either gate speculative decoding on current queue depth, or simply disable it at the serving level when avg batch size exceeds a threshold. vLLM's scheduler can be configured to disable spec decoding when the running batch exceeds a configured size.

Draft model OOM. Adding a 3B draft to a system already squeezed for KV cache memory causes out-of-memory crashes under load. This is the KV cache math problem in disguise. The 6 GB for the draft comes directly out of space that would have been KV cache, reducing max concurrent context or batch size. EAGLE-2 self-speculative avoids this entirely since it adds only a small auxiliary head, not a full separate model.

Greedy decoding gotcha. At temperature=0.0 (greedy), acceptance tops out at the draft/target argmax agreement rate — don't expect 1.0, and don't treat sub-1.0 acceptance as a bug. A separate, subtler issue is reproducibility: at temperature=0 the output of a speculative run can differ from a target-only run, because the batched verification pass and an incremental decode pass take different numerical paths through attention, and small floating-point differences can flip near-tied token rankings. If your application requires exact reproducibility at temperature=0, test this carefully. Most production systems use temperature > 0 where this doesn't matter.

Slow TTFT gets worse. Speculative decoding only helps decode, not prefill. If your prefill phase is already your bottleneck (long system prompts, large RAG contexts), adding speculative decoding to the decode phase has no impact on TTFT. Diagnose the bottleneck before adding complexity. If TTFT is the problem, chunked prefill or disaggregated prefill is the answer; speculative decoding is not.

Output length sensitivity. Speculative decoding's speedup is proportional to total decode length. For very short responses (1-10 tokens), the setup cost of running the draft model and the verification pass eats into the gains. On a 200-token response the speedup is close to the theoretical maximum; on a 5-token response it may be near-zero. If your application generates short bursts, profile actual end-to-end latency, not just tokens/second.

When to use what

flowchart TD
    Q1{"Single-user or\nlow batch size\n≤ 16 requests?"}
    Q1 -->|Yes| Q2{"Can you run\na separate\ndraft model?"}
    Q1 -->|No| N1["Disable or skip spec decoding\nFocus on continuous batching\nand quantization instead"]

    Q2 -->|"Yes — 6GB VRAM to spare"| Q3{"Traffic is\nhomogeneous\n(chat, code, etc.)?"}
    Q2 -->|"No — VRAM constrained"| SE["EAGLE-2 self-speculative\n(no separate model overhead)\nSGLang default"]

    Q3 -->|Yes| CD["Classic spec decoding\nSame-family draft model\nK=5, BF16 draft"]
    Q3 -->|"No — mixed workloads"| SE

    CD --> MON["Monitor acceptance rate\nAlert below 0.55"]
    SE --> MON

    style N1 fill:#ff2e88,color:#fff
    style SE fill:#0e7490,color:#fff
    style CD fill:#15803d,color:#fff
    style MON fill:#ffaa00,color:#0a0a0f

The practical decision is straightforward. If you're serving at low concurrency and latency-per-user is your primary metric — interactive chat, coding assistant, API with tight p95 SLAs — speculative decoding is worth enabling. Start with SGLang's EAGLE-2 default if you're running a supported model; it gives higher acceptance rates with less operational overhead than a separate draft. If you're on vLLM and have a well-matched same-family draft available, the classic approach is solid. And if your traffic is RAG or code editing, try n-gram speculation first — it costs nothing to enable and nothing when it misses.

If you're optimizing for throughput — batch inference, offline document processing, high-concurrency serving — speculative decoding likely hurts. Your optimization surface is continuous batching, quantization, and KV cache efficiency.

The cleanest production configuration is to measure your typical concurrent batch size. Below 8: enable speculative decoding unconditionally. Between 8 and 32: A/B test it with acceptance-rate monitoring. Above 32: leave it off unless you have specific p95 latency requirements that throughput-oriented optimizations can't meet.

One final note on the guarantee that makes all of this safe to ship: the math really does hold. The output distribution is identical to the target model. You're not trading quality for speed the way quantization does. You're trading extra compute (the draft pass) for parallelism inside the target's verification pass. When the acceptance rate is high enough, that trade is clearly in your favor. When it isn't, the metric tells you. That's a rare combination in inference optimization — a technique with a hard theoretical guarantee and a real-time signal that tells you when to turn it off.