Streaming, Batching, and Throughput: Latency vs Cost Trade-offs in Production

A team at a startup I know was paying $18,000 a month for LLM serving — provider API calls plus a self-hosted chat deployment. Their back-of-envelope said it should be $6,000. The culprit turned out to be three things: their nightly document-summarisation pipeline hitting real-time endpoints, their chat serving endpoint running batch size 1, and their retry logic on failed streams re-sending partial completions that had already been delivered to users. None of it was a model problem. None of it required changing a prompt. It was all serving configuration.

This article is about the three axes — streaming, batching, and async batch processing — and how they interact with latency, throughput, and cost. They are not alternatives; they belong at different layers of the same system.

What streaming actually does (and does not do)

Server-Sent Events (SSE) is the standard transport for LLM streaming. The model generates tokens one at a time (or in small groups depending on the serving engine), and each token is flushed to the client immediately rather than accumulating in a buffer until the response is complete.

import anthropic

client = anthropic.Anthropic()

with client.messages.stream(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain KV cache in 200 words."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

The user sees the first token in roughly the same time it takes the model to generate it — time-to-first-token (TTFT). For a GPT-4-class model at a major provider, this is typically 200–800ms depending on load. For a local deployment on an H100, it can be under 100ms at low batch sizes. The rest of the response streams in at whatever the model's decode speed is.

What streaming does not change: the total number of input tokens the provider ingests, the total number of output tokens generated, the amount of GPU compute consumed, or your bill. Every major provider prices on token count. Streaming is a delivery mechanism, not a cost reduction mechanism.

The UX gain is real and significant. A response that takes 8 seconds to generate feels tolerable when you see the first word in 300ms and text is flowing continuously. The same response buffered feels like an 8-second hang. This is why all conversational products stream.

sequenceDiagram
    participant U as User
    participant API as LLM API
    participant GPU as GPU (generating)

    U->>API: POST /messages (stream=true)
    API->>GPU: start generation
    GPU-->>API: token 1
    API-->>U: data: token 1 (SSE)
    GPU-->>API: token 2
    API-->>U: data: token 2 (SSE)
    GPU-->>API: tokens 3–47 (bulk)
    API-->>U: data: tokens 3–47
    GPU-->>API: [DONE]
    API-->>U: data: [DONE]
    Note over U,GPU: Bill = same as non-streaming

One thing streaming does affect: how you measure latency in your observability stack. TTFT and time-per-output-token (TPOT) are the two metrics that matter for interactive quality. TTFT is the wall-clock time from request send to first token received. TPOT is the average gap between tokens during generation. A degraded TPOT (tokens arriving in bursts with gaps) creates a stuttering effect that users notice. Track both separately; end-to-end latency alone hides which phase broke.

Continuous batching: how serving engines multiply throughput

Static batching — the naive approach — takes a group of requests, runs them all through the model together in a single forward pass, waits for every sequence in the batch to finish, then starts the next batch. The problem is that sequences finish at wildly different lengths. A batch of 16 requests where one generates 800 tokens and fifteen generate 50 tokens means fifteen sequences are sitting idle waiting for the long one to finish. GPU utilisation collapses.

Continuous batching (sometimes called iteration-level scheduling) solves this by treating each token generation step as a scheduling opportunity. When a sequence finishes, the serving engine immediately inserts a new request into the batch at that slot. No waiting for the rest to finish.

{ "type": "batching", "title": "Static vs continuous batching: GPU utilisation" }

vLLM and SGLang both implement continuous batching. SGLang v0.4.3 benchmarks at 16,200 tokens/second on an H100 for standard workloads; Morph Compact reaches 33,000 tokens/second for context-compacted inputs (as of early 2026, exact numbers shift with updates — treat these as order-of-magnitude calibration). Note that the second number is not an apples-to-apples engine comparison — the inputs are compacted, so the 2x gap measures engine choice plus input compaction together. Still, the stack you serve with is a real lever before you touch model or prompts.

PagedAttention compounds this. The KV cache — the memory structure holding attention keys and values for every token a sequence has processed — has to live on GPU during generation. In static allocation, you reserve the maximum possible KV cache for each sequence up front, even though most sequences use a fraction of it. Waste ranges from 30% to 90% of reserved memory in practice.

PagedAttention breaks the KV cache into fixed-size pages (typically 16 or 32 tokens per block) and allocates pages on demand, like virtual memory in an OS. A sequence that generates 50 tokens uses 50 tokens of KV cache, not 4096. The reclaimed memory fits more parallel sequences. vLLM's original paper reported up to 24x higher serving throughput versus Hugging Face Transformers on the same hardware, primarily from this effect.

For the KV cache math in depth, see the KV cache GPU memory article. For a detailed look at how prefill and decode phases split that compute budget, see Prefill vs Decode.

The batch size dial and why it breaks interactive latency

Batch size is the number of sequences being processed simultaneously. Higher batch size = more GPU FLOP utilisation = higher tokens-per-second throughput. But it has a cost: queueing delay.

Even under continuous batching, a new request has to wait twice. First for a slot: batch capacity is bounded by KV cache memory, and when every block is held by in-flight sequences, the new arrival queues until one finishes or gets preempted. Then for prefill: its prompt has to be scheduled against the decode steps of everything already generating. And each individual decode step gets slower as the batch grows, because the GPU is doing more work per iteration. Stack those three effects and TTFT climbs with batch size even though no request ever waits for another's full completion.

Concrete numbers (illustrative, H100 serving a 70B model, mid-2026):

Batch size 1:
  TTFT: ~80ms
  Throughput: ~150 tok/s

Batch size 8:
  TTFT: ~200ms
  Throughput: ~800 tok/s

Batch size 32:
  TTFT: ~600ms
  Throughput: ~2,400 tok/s

Batch size 128:
  TTFT: ~2,400ms (above most interactive SLOs)
  Throughput: ~6,000 tok/s

→ For p99 TTFT < 500ms: max batch size 8–16
→ For throughput-optimised background jobs: batch size 64–256

This is why the critical separation is workload class. Interactive chat needs TTFT under 300ms — which caps batch size. Background summarisation does not care about TTFT at all; it needs maximum throughput at minimum cost. They cannot share a serving configuration without one destroying the other.

The standard architecture is two pools: a streaming-optimised pool with small batch sizes and autoscaling triggered on TTFT p99, and a throughput-optimised pool with large batch sizes and autoscaling triggered on queue depth. Route at the application layer by whether the caller is blocking on the result.

If you run a single shared deployment, there is a built-in mitigation to try before buying a second pool: chunked prefill. Instead of running a long prompt's prefill as one monolithic pass that stalls every in-flight stream, the engine splits it into chunks and interleaves them with ongoing decode steps — a 50K-token arrival costs everyone a few slightly slower iterations instead of a multi-second freeze. It is on by default in current vLLM, and the trade is explicit: the big request's TTFT gets a little worse, everyone else's TPOT stays stable. At larger fleet sizes the heavyweight version of the same idea is prefill/decode disaggregation — dedicated prefill GPUs stream KV caches to decode GPUs so the two phases never compete for the same compute; Prefill vs Decode covers the mechanics.

Async Batch API: the 50% discount most teams leave on the table

Both OpenAI and Anthropic offer async Batch APIs that process requests asynchronously — typically within 24 hours — at approximately 50% of the real-time API cost. The mechanism is simple: providers can route batch jobs to spare capacity and off-peak compute, then they share the savings.

import anthropic
import json

client = anthropic.Anthropic()

# Prepare a batch of summarisation requests
requests = [
    {
        "custom_id": f"doc-{i}",
        "params": {
            "model": "claude-sonnet-4-5",
            "max_tokens": 256,
            "messages": [
                {"role": "user", "content": f"Summarise in 3 sentences: {doc_text}"}
            ],
        },
    }
    for i, doc_text in enumerate(documents)
]

# Submit batch
batch = client.messages.batches.create(requests=requests)
print(f"Batch ID: {batch.id}")  # poll or webhook for results

The workloads that belong here are any workload where the caller is not blocking on the result:

• Document summarisation pipelines
• Classification and labelling at scale
• Embedding generation (though dedicated embedding APIs are often cheaper still)
• Nightly evaluation runs against golden datasets
• Data enrichment jobs upstream of a RAG index build

Using a real-time streaming endpoint for any of these is a budget leak. You pay full price, compete with interactive users for provider capacity during peak hours, and get no benefit from the speed since the job is running asynchronously anyway. The async Batch API is the right default for non-interactive work.

The constraint is SLA: if you need results within minutes, async batch does not work. The right threshold depends on your workflow, but anything with a >60-minute tolerance is a candidate.

See Token Economics for how the input/output pricing asymmetry changes the arithmetic here — output tokens are 3–5x more expensive than input tokens, so the 50% batch discount on a generation-heavy job saves more in absolute terms than on a classification job with short outputs.

Speculative decoding: a brief note on another throughput lever

Speculative decoding runs a small draft model to propose k tokens ahead, then verifies them in a single forward pass of the large target model. When tokens are accepted, you get k tokens for the cost of one target-model pass. This delivers 2–5x speedup for certain workloads without changing output quality.

It belongs in the serving engine configuration, not the application layer, so most teams using provider APIs do not control it. Self-hosting teams using vLLM or SGLang can enable it. EAGLE3 speculative decoding reached up to 5x speedup in domain-specific production deployments as of early 2026.

{ "type": "speculative", "title": "Speculative decoding: draft → verify → accept/reject" }

For the mechanics in depth, see Speculative Decoding.

What breaks

Streaming retries send corrupted output

This is the most operationally damaging failure mode and the one teams discover last. A stream fails at token 47 out of a 200-token response. Your retry middleware fires and starts a fresh request. The new request generates a different completion from token 1. The user's UI has already rendered tokens 1–47 from the first attempt.

Result: the user sees tokens 1–47 (first generation) immediately followed by a different tokens 1–200 (second generation) — a doubled, incoherent response.

Safe handling has two branches. If no tokens have been rendered yet (the failure occurred before the UI displayed anything), retry is transparent. If tokens were rendered, you must either: (a) surface the failure to the user with an explicit error state, or (b) have the UI clear the partial content and render the full new response once complete. Neither is elegant, but they are both honest. Silent retry is not.

def stream_with_retry(client, request_params, max_retries=2):
    tokens_sent = []
    
    for attempt in range(max_retries):
        try:
            with client.messages.stream(**request_params) as stream:
                for text in stream.text_stream:
                    tokens_sent.append(text)
                    yield text  # send to client
            return  # success
        except Exception as e:
            if tokens_sent and attempt < max_retries - 1:
                # Partial content was sent — do NOT silently retry
                # Either surface error or yield a sentinel for UI reset
                yield "\n\n[Connection interrupted — please refresh]"
                return
            elif not tokens_sent and attempt < max_retries - 1:
                tokens_sent = []  # reset for clean retry
                continue
            raise

Large batches killing interactive p99

A single long-running generation sequence at the head of a large batch will stall TTFT for every request waiting to enter the batch. This shows up in your metrics as occasional p99 spikes that look random but correlate with long-document processing or code generation requests entering the serving pool.

Fix: separate pools, or configure priority queuing if your serving engine supports it (SGLang has scheduling policies; vLLM supports preemption in recent versions). Interactive requests preempt or wait in a separate high-priority queue.

Context window mismatch during fallback

If you route failed requests to a fallback model with a smaller context window — say, a 32K context model as fallback to a 128K primary — a request carrying 50K tokens of conversation history will silently truncate or error. This is covered in depth in the Fallbacks, Retries, and Circuit Breakers article, but it is worth naming here: the serving configuration and the fallback chain must be designed together, not independently.

Misidentifying TTFT degradation as model slowdown

TTFT increases from three distinct causes: increased batch size on the server (more requests queued), increased input token length (prefill phase takes longer), and model capacity pressure (provider rate limiting or autoscaling lag). All three look identical in end-to-end latency metrics. Track TTFT and TPOT separately. If TPOT is stable but TTFT is rising, the problem is batch queue depth or prefill length, not decode throughput.

flowchart LR
    TTFT["TTFT rising"] --> CAUSE1["Batch queue depth\n→ check server utilisation"]
    TTFT --> CAUSE2["Input token length increasing\n→ check context growth per session"]
    TTFT --> CAUSE3["Provider rate limiting\n→ check 429 rate and headers"]
    TPOT["TPOT rising"] --> CAUSE4["Decode throughput degraded\n→ memory pressure or model load"]
    style TTFT fill:#00e5ff,color:#0a0a0f
    style TPOT fill:#a855f7,color:#fff

The decision in practice

The serving configuration decision follows from answering three questions about each workload:

1. Does the caller block on the result?

If yes (interactive chat, search, real-time classification): use streaming. Configure a small batch size that keeps TTFT under your SLO. Track TTFT and TPOT in your observability stack — LLM observability tooling covers what to instrument.

If no (summarisation, eval, data enrichment): use the async Batch API. Do not touch a streaming endpoint for this.

2. Are you self-hosting?

If yes: use continuous batching via vLLM or SGLang. Enable PagedAttention (it is on by default in both). Set batch size separately for interactive and background pools. Watch the Serving Engine Comparison article for the current state of vLLM vs SGLang — the performance gap shifts with each release.

If no (provider API): your batching configuration is the provider's problem. Your levers are model choice, request structure, prompt caching, and routing — covered in Model Routing and Cascades.

3. Is throughput or cost the binding constraint?

Both usually matter, but the bottleneck changes as you scale. At low volume, provider API cost dominates — batch API discounts and prompt caching are the first levers. Above roughly $4,200/month in API spend (illustrative mid-2026 threshold), self-hosting becomes worth evaluating, as covered in Build vs Buy.

The table below maps workload type to configuration:

Workload	Endpoint	Batch size	TTFT target	Cost target
Interactive chat	Streaming	4–8	<300ms p50	Full price — UX is the product
Real-time classification	Streaming or sync	8–16	<500ms p99	Full price
Document summarisation	Async Batch API	N/A (provider handles)	Hours	~50% discount
Nightly eval runs	Async Batch API	N/A	Hours	~50% discount
Self-hosted interactive	Continuous batching	8–16	<300ms p50	Hardware + ops
Self-hosted background	Continuous batching	64–256	Doesn't matter	Maximise tok/s

The $18,000 bill that should have been $6,000? The nightly summarisation pipeline moved to the Batch API ($4,000/month saved). Chat went from batch size 1 to continuous batching on a properly configured vLLM deployment ($5,000/month saved in GPU infrastructure). The retry middleware — which had been silently re-running full completions on every mid-stream failure and billing the duplicate tokens — was fixed in a single afternoon ($3,000/month saved). Twelve thousand dollars a month, back to the $6,000 estimate. No model changes. No quality regressions. The work is in the plumbing.