Streaming LLM Responses: The Engineering Nobody Talks About

The support ticket came in three weeks after launch: "Your AI keeps loading forever, then shows nothing." The logs showed the completion succeeded — 800 tokens, 6.2 seconds, response code 200. What the logs did not show was that the client had navigated away at second two. The server kept generating. The stream flushed to a closed socket. Tokens billed, nothing delivered. At 40,000 requests per day, that was a meaningful fraction of the API bill going to users who had already left.

Streaming looks like a solved problem until you run it in production. Then it turns into four distinct engineering problems — transport selection, partial-JSON parsing, backpressure, and cancellation — that nobody covers in the "getting started" documentation.

Why streaming exists and what it actually changes

When a user sends a message, the model first processes all input tokens in the prefill phase (compute-bound, but parallel and fast), then generates output tokens one at a time in the decode phase (memory-bandwidth-bound and slow — every token reads the full weights and KV cache). On a 500-token reply at 80 tokens/second, the decode phase alone takes six seconds. Without streaming, the user waits for all six seconds before seeing anything.

Streaming sends each token over the wire the moment it is generated. The time-to-first-token (TTFT) is the prefill time — typically 200-600ms on current hosted APIs. The user sees the first word in under a second and reads the response as it generates. Total wall time is identical. Compute cost is identical. What changes is perceived latency and the ability to abort.

That last point matters economically. A user who sees a response going wrong — the model is writing in the wrong language, or answering a different question — can hit Stop after 50 tokens instead of waiting for 800. If your app wires cancellation correctly, you stop generating and billing at 50 tokens, not 800.

sequenceDiagram
    participant U as User
    participant A as Your App Server
    participant P as LLM Provider

    U->>A: POST /chat
    A->>P: POST /completions (stream: true)
    Note over P: Prefill phase: 300ms
    P-->>A: data: {"delta": "The "}
    A-->>U: chunk: "The "
    P-->>A: data: {"delta": "answer "}
    A-->>U: chunk: "answer "
    P-->>A: data: {"delta": "is..."}
    A-->>U: chunk: "is..."
    Note over U: User reads while generating
    P-->>A: data: [DONE]
    A-->>U: stream end

SSE vs WebSockets: pick the right transport

Server-Sent Events are the right choice for LLM streaming in the overwhelming majority of cases. SSE is a standard browser API (EventSource) that opens a persistent HTTP/1.1 or HTTP/2 connection and receives data: lines pushed from the server. It is unidirectional by design. It has built-in reconnection logic using Last-Event-ID. Because it is plain HTTP, no infrastructure needs a protocol upgrade — though, as the What breaks section covers, buffering proxies and CDNs still need explicit configuration to pass text/event-stream through unbuffered. The LLM completion pattern (one request in, a stream of tokens out) maps onto SSE naturally.

One honest caveat about that built-in reconnection: it belongs to the browser's EventSource API, and EventSource can only issue GET requests with no body. LLM chat needs a POST with a message payload, so every real chat client — including the samples in this article — uses fetch with a ReadableStream instead, and forfeits automatic reconnection. You handle reconnection at the application layer, and there is less to it than it sounds: provider completion streams are not resumable, so a dropped connection mid-generation cannot pick up where it left off. The retry rule later in this article (retry only before the first byte) is the whole story.

WebSockets are appropriate when you need bidirectional streaming that genuinely cannot be modeled as a series of request-response pairs. Voice pipelines are the clearest example: microphone audio streams in continuously while synthesized audio streams back, and you want to interrupt mid-utterance. A voice pipeline with VAD and turn detection has fundamentally different transport requirements than a chat completion stream. WebSockets also suit collaborative real-time UIs where multiple clients push and receive edits simultaneously.

For anything else — chat, document summarization, code generation, agent step narration — SSE is simpler, more compatible, and no slower.

The raw SSE wire format from OpenAI and Anthropic looks like this:

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":"The "},"index":0}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":"answer "},"index":0}]}

data: [DONE]

Each line is prefixed with data: , followed by a JSON chunk. The stream ends with data: [DONE]. The server sets Content-Type: text/event-stream and keeps the connection alive. The browser's EventSource handles reconnection if the connection drops, using the Last-Event-ID header if you set event IDs.

A minimal streaming endpoint in Python using the Anthropic SDK:

import json

import anthropic
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

client = anthropic.Anthropic()
app = FastAPI()

@app.post("/chat")
async def chat(body: dict):
    def generate():  # sync generator: the sync client would block the event loop inside async def
        with client.messages.stream(
            model="claude-sonnet-4-5",
            max_tokens=1024,
            messages=[{"role": "user", "content": body["message"]}],
        ) as stream:
            for text in stream.text_stream:
                yield f"data: {json.dumps({'delta': text})}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(generate(), media_type="text/event-stream")

And the equivalent with the OpenAI SDK, showing the raw chunk structure:

from collections.abc import Iterator

from openai import OpenAI

client = OpenAI()

def stream_completion(messages: list[dict]) -> Iterator[str]:
    with client.chat.completions.create(
        model="gpt-4o",
        messages=messages,
        stream=True,
    ) as stream:
        for chunk in stream:
            delta = chunk.choices[0].delta
            if delta.content:
                yield delta.content
            if chunk.choices[0].finish_reason == "stop":
                break

Partial-JSON parsing for streamed tool calls

This is where most streaming implementations break silently.

When a model calls a tool, the function arguments arrive as a JSON string that is fragmented across multiple delta events. Each chunk contains a string slice — not a valid JSON object, just a piece of one:

data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"name":"search","arguments":""}}]}}]}
data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"query\":"}}]}}]}
data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"\"weather in "}}]}}]}
data: {"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"Paris\"}"}}]}}]}
data: {"choices":[{"finish_reason":"tool_calls"}]}

The argument string {"query": "weather in Paris"} arrives across four chunks. None of the intermediate fragments is valid JSON. JSON.parse('{"query":') throws. JSON.parse('"weather in ') throws. You must buffer all argument deltas per tool-call index and parse only after finish_reason signals completion.

from collections import defaultdict
import json

def collect_tool_calls(stream) -> list[dict]:
    """Buffer streamed tool call deltas and return complete calls."""
    tool_calls_buffer: dict[int, dict] = defaultdict(
        lambda: {"id": "", "name": "", "arguments": ""}
    )

    for chunk in stream:
        choice = chunk.choices[0]
        delta = choice.delta

        if delta.tool_calls:
            for tc_delta in delta.tool_calls:
                idx = tc_delta.index
                if tc_delta.id:
                    tool_calls_buffer[idx]["id"] = tc_delta.id
                if tc_delta.function.name:
                    tool_calls_buffer[idx]["name"] = tc_delta.function.name
                if tc_delta.function.arguments:
                    tool_calls_buffer[idx]["arguments"] += tc_delta.function.arguments

        if choice.finish_reason == "tool_calls":
            break

    # Only parse complete JSON once buffering is done
    return [
        {
            "id": tc["id"],
            "name": tc["name"],
            "arguments": json.loads(tc["arguments"]),  # Safe: string is now complete
        }
        for tc in tool_calls_buffer.values()
    ]

If you need to surface argument fields to the UI while they are streaming — useful for showing the tool call in progress — you need an incremental JSON parser. The Anthropic Python SDK ships one natively in its streaming helpers. For OpenAI, libraries like partial-json (npm) or ijson (Python) handle incremental parsing without throwing on incomplete input. The key insight is that you are not parsing JSON on each raw delta; you are feeding a stateful parser that emits complete values as their final delimiter arrives.

{ "type": "agent-loop", "scenario": "weather-trip", "title": "Where tool-call JSON sits in the agent loop — the arguments below arrive as the fragments shown above" }

Anthropic's streaming protocol differs slightly from OpenAI's — it uses event: type lines and nests deltas differently. The Anthropic SDK's stream.text_stream and stream.input_json_stream helpers handle this correctly and are the right abstraction to use rather than parsing raw SSE yourself.

Backpressure: the OOM bug that hides until load

In the browser, SSE backpressure is invisible. The browser buffers internally and slows down event dispatch; you never hit OOM because browser tabs have hard memory limits and the browser manages it.

In a Node.js relay server — an app server that receives SSE from the provider and forwards it to connected clients — backpressure is a real problem. If the downstream client (your browser user, or another service) consumes chunks slower than the provider emits them, the Node.js Readable stream buffers the excess in memory. Under normal conditions this stays small. Under load, with many concurrent streams and some slow clients, it grows unbounded.

The fix is to await each write before reading the next chunk. If you are using Express with res.write():

// WRONG: fire-and-forget writes, no backpressure handling
for await (const chunk of stream) {
  res.write(`data: ${JSON.stringify(chunk)}\n\n`);
}

// CORRECT: await the drain event if write() returns false
async function writeWithBackpressure(
  res: express.Response,
  data: string
): Promise<void> {
  const ok = res.write(data);
  if (!ok) {
    // The write buffer is full; wait for it to drain before continuing
    await new Promise<void>((resolve) => res.once("drain", resolve));
  }
}

for await (const chunk of stream) {
  await writeWithBackpressure(res, `data: ${JSON.stringify(chunk)}\n\n`);
}

In practice, the simplest approach is to use Node.js pipeline() with a Transform stream between the provider stream and the response. pipeline() handles backpressure correctly by design and also cleans up all streams on error or disconnect.

import { pipeline } from "node:stream/promises";
import { Transform } from "node:stream";

const transform = new Transform({
  transform(chunk, _encoding, callback) {
    // chunk is a Buffer from the provider's SSE stream
    this.push(`data: ${chunk.toString()}\n\n`);
    callback();
  },
});

await pipeline(providerStream, transform, res);

The pipeline() call resolves when the stream ends and rejects — cleaning up all streams — on any error or if the destination closes early. This is the right primitive for relay servers.

Cancellation: stop paying for tokens nobody receives

Client disconnect cancellation is the highest-ROI fix in most streaming codebases. Without it:

1. User sends a message, streaming begins.
2. User navigates away or closes the tab.
3. Your relay server's TCP connection to the client closes.
4. Your relay server continues reading from the provider stream.
5. The provider continues generating.
6. You pay for every token generated.
7. Nothing is delivered to anyone.

At scale, this is not a corner case. Mobile users navigate away constantly. Long-running generations on slow connections drop frequently. Even desktop users abort and retype. A 5-10% abort rate on 500-token responses means 25-50 wasted tokens per request averaged across all traffic — and hundreds of wasted tokens on each aborted request — which adds up at volume.

The fix is to listen for the client disconnect event and abort the upstream request:

app.post("/chat", async (req, res) => {
  const controller = new AbortController();

  // Fire when client disconnects (TCP close or explicit abort)
  req.on("close", () => {
    controller.abort();
  });

  try {
    const stream = await openai.chat.completions.create(
      {
        model: "gpt-4o",
        messages: req.body.messages,
        stream: true,
      },
      {
        signal: controller.signal, // AbortController propagates to the fetch call
      }
    );

    res.setHeader("Content-Type", "text/event-stream");
    res.setHeader("Cache-Control", "no-cache");

    for await (const chunk of stream) {
      if (res.writableEnded) break; // Double-check: client already gone
      const text = chunk.choices[0]?.delta?.content ?? "";
      if (text) res.write(`data: ${JSON.stringify({ text })}\n\n`);
    }
  } catch (err) {
    if (err.name === "AbortError") {
      // Client disconnected; this is expected, not an error
      return;
    }
    throw err;
  } finally {
    res.end();
  }
});

The Anthropic SDK's stream object exposes stream.controller — calling stream.controller.abort() on client disconnect stops the upstream HTTP request immediately. Tokens stop being generated. Billing stops.

sequenceDiagram
    participant U as User
    participant R as Relay Server
    participant P as LLM Provider

    U->>R: POST /chat
    R->>P: stream request
    P-->>R: token 1
    R-->>U: chunk 1
    P-->>R: token 2
    R-->>U: chunk 2
    Note over U: User navigates away
    U--xR: TCP close
    Note over R: req "close" event fires
    R->>P: abort() — stop generating
    Note over P: Generation stops, billing stops

On the client side in the browser, the pattern mirrors this:

let controller: AbortController | null = null;

async function sendMessage(message: string) {
  controller = new AbortController();

  const response = await fetch("/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ message }),
    signal: controller.signal,
  });

  const reader = response.body!.getReader();
  const decoder = new TextDecoder();
  let pending = ""; // carries partial events between reads

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    pending += decoder.decode(value, { stream: true });

    // Reader chunks do not align with SSE event boundaries.
    // Split on the event delimiter and keep the trailing partial event.
    const events = pending.split("\n\n");
    pending = events.pop() ?? "";
    for (const event of events) {
      processEvent(event); // parse "data: {...}" lines, update UI
    }
  }
}

function stopGeneration() {
  controller?.abort();
  controller = null;
}

Note the line buffering. reader.read() hands you whatever bytes arrived on the socket, and those chunks do not align with SSE event boundaries — a data: line, or the JSON inside it, can be split across two reads. Feeding raw chunks straight to a parser is the single most common client-side streaming bug. Accumulate decoded text, split on \n\n, process only complete events, and carry the remainder into the next read.

Calling controller.abort() closes the fetch connection. The relay server's req.on("close") handler fires. The upstream provider request is aborted. Clean shutdown at every layer.

Rendering streamed markdown without flicker

Streaming plain text is straightforward — append each token to the DOM and you are done. Streaming markdown is not, because markdown is not a streaming format. Markers like **bold**, `code`, and ```python are delimiters that appear as raw characters mid-stream before their closing counterpart arrives.

If you re-render on every token delta, the user sees:

"The answer is **im" → renders as: The answer is **im
"portant" → re-renders: The answer is **important"  → still broken
"**" → re-renders: The answer is important  ← flicker resolves

The bold text appears broken for the entire duration of the marked span. On fast connections with many tokens per second, this flicker is continuous and distracting.

Three approaches, in order of implementation complexity:

Option 1: Two-phase rendering. Render incoming deltas as plain text while streaming; do a full markdown parse-and-replace on the completed string. Simple, no flicker. The downside is that code blocks and headers stay as raw text until the stream ends. For most completions this is fine.

Option 2: Debounced incremental parsing. Buffer deltas for 50-100ms (or N tokens, or until a newline), then re-render the accumulated string. This reduces flicker to a 50-100ms lag and handles most delimiters correctly because by the time you re-render, most short spans have closed.

let buffer = "";
let renderTimer: ReturnType<typeof setTimeout> | null = null;

function onToken(token: string) {
  buffer += token;

  // Debounce: only re-render after 80ms of silence or at a newline boundary
  if (renderTimer) clearTimeout(renderTimer);
  if (token.includes("\n")) {
    renderMarkdown(buffer); // Render immediately at line boundaries
  } else {
    renderTimer = setTimeout(() => renderMarkdown(buffer), 80);
  }
}

Option 3: Streaming-aware markdown parser. Libraries like marked with walkTokens hooks or micromark can parse incomplete input and emit partial tokens. This is the most correct approach but also the most complex to integrate with React's reconciler without unnecessary re-renders.

In React, option 2 is typically the right balance. Use a useRef for the accumulated string (no re-render on each token) and useState for the rendered version that actually drives the DOM, only set from the debounce timer:

function StreamingMessage({ stream }: { stream: AsyncIterable<string> }) {
  const [renderedHtml, setRenderedHtml] = useState("");
  const bufferRef = useRef("");

  useEffect(() => {
    let timer: ReturnType<typeof setTimeout> | null = null;

    async function consume() {
      for await (const token of stream) {
        bufferRef.current += token;

        if (timer) clearTimeout(timer);
        if (token.includes("\n")) {
          setRenderedHtml(marked.parse(bufferRef.current));
        } else {
          timer = setTimeout(() => {
            setRenderedHtml(marked.parse(bufferRef.current));
          }, 80);
        }
      }
      // Final render on stream end
      if (timer) clearTimeout(timer);
      setRenderedHtml(marked.parse(bufferRef.current));
    }

    consume();
  }, [stream]);

  return (
    <div
      className="prose"
      dangerouslySetInnerHTML={{ __html: renderedHtml }}
    />
  );
}

The dangerouslySetInnerHTML here is acceptable because the HTML source is marked.parse() output from model text, not user-controlled markup. Sanitize with DOMPurify if your application renders user-submitted content through this path.

What breaks

flowchart TD
    S[Streaming request] --> B{Where does it break?}
    B --> C1[Client disconnect\nnot wired → billing waste]
    B --> C2[Tool call JSON.parse on\npartial delta → throws]
    B --> C3[No backpressure in relay\n→ OOM under load]
    B --> C4[Markdown re-render\non every token → flicker]
    B --> C5[Retry after partial\ndelivery → duplicate content]
    B --> C6[Stream timeout too short\n→ connection dropped mid-reply]

    style C1 fill:#ff2e88,color:#111
    style C2 fill:#ff2e88,color:#111
    style C3 fill:#ff2e88,color:#111
    style C4 fill:#ffaa00,color:#0a0a0f
    style C5 fill:#ff2e88,color:#111
    style C6 fill:#ffaa00,color:#0a0a0f

Client disconnect not wired. The most common and most expensive failure. Providers keep generating; you keep paying. Fix: req.on("close") + AbortController.

Retrying a stream after partial delivery. If your relay sends 200 tokens to the user and then the upstream provider connection drops, a naive retry fires a new request. The model generates a fresh 500-token completion — usually with different content. The user sees the stream start over, or worse, the two completions get concatenated. Streaming retries are fundamentally different from non-streaming retries: you cannot transparently retry once any content has been sent. The fallbacks and retries article covers the right pattern (retry only before first byte, degrade to non-streaming on failure). This problem is distinct from the general reliability concerns; stream-specific retry logic must be layered on top.

Stream timeout misconfiguration. Long streaming responses — 1,000+ tokens — take 15-20 seconds on slower models. Default HTTP timeouts of 30 seconds in many frameworks will cut these short. Nginx's proxy_read_timeout (default 60 seconds) is an inactivity timeout between successive reads, not a total-duration cap — a stream emitting tokens steadily never trips it, however long it runs. What does trip it is an idle gap mid-stream: a tool-execution pause, a provider stall, a long think before the next token. Two fixes, use both: raise the timeout to cover your worst expected stall, and send periodic SSE heartbeat comments (a line starting with :, e.g. : ping, every 15-30 seconds) so intermediaries see traffic even while generation is quiet.

Proxy and CDN buffering. Some CDNs buffer the entire response body before forwarding it to the client, eliminating the streaming benefit entirely. Cloudflare's default behavior with certain cache settings does this. Check your headers: X-Accel-Buffering: no for Nginx, and ensure your CDN is configured to pass through text/event-stream without buffering.

Server-Sent Events and HTTP/2 multiplexing. SSE over HTTP/1.1 holds one TCP connection open per stream. Browsers limit concurrent HTTP/1.1 connections to six per origin — which means only six concurrent SSE streams before new requests queue. HTTP/2 multiplexes many streams over one connection, lifting this constraint. Most modern stacks handle HTTP/2 SSE correctly, but if you are seeing mysterious request queuing for a user with multiple open chat windows, this is the likely cause.

Context window and token cost interaction. Streaming does not change what you pay for, but it can mask token bloat. A non-streaming call with a 5-second wait makes slow responses obvious and prompts optimization. A streaming call that shows content immediately feels fast even if it is generating 1,200 tokens for a question that needs 150. Monitor total token counts per completion, not just TTFT. See Token Economics for the full cost breakdown.

The numbers: what streaming actually buys you

Production scenario: chat assistant, GPT-4o, ~400 tokens average response

Without streaming:
  TTFT:                          400ms (prefill)
  Total generation time:         400ms + (400 tokens / 90 tok/s) = 4.9s
  User sees first content at:    4.9s
  User frustration threshold:    ~3s (Nielsen: 10s is the limit; 3s is where engagement drops)

With streaming:
  User sees first token at:      ~500ms
  User is reading at:            500ms
  Full response complete at:     4.9s (identical)
  Perceived completion:          much shorter — user is engaged from 500ms

Cancellation savings (10% abort rate, user cancels at token 80 of 400):
  Without cancellation: 400 tokens billed per abort
  With cancellation:    80 tokens billed per abort
  Tokens saved per abort: 320
  At 100,000 req/day with 10% abort rate: 10,000 aborts × 320 tokens = 3.2M tokens/day saved
  At $5/M output tokens: ~$16/day saved, ~$5,800/year
  At higher volumes, this scales linearly

The cancellation savings are real but modest at this volume. At 1M requests/day they become meaningful. More importantly, wiring cancellation is half a day of engineering work; the argument for doing it is correctness as much as cost.

{ "type": "token-cost", "title": "Token cost breakdown — streaming does not reduce cost but cancellation does" }

Latency budget across the full streaming path

At a systems level, streaming TTFT depends on every hop in the chain. Each hop can add buffering:

sequenceDiagram
    participant B as Browser
    participant E as Edge/CDN
    participant A as App Server
    participant P as LLM Provider

    B->>E: POST /chat
    E->>A: forward (may buffer)
    A->>P: POST /completions stream:true
    Note over P: Prefill: 300-600ms
    P-->>A: first token chunk
    Note over A: Process delta, encode SSE
    A-->>E: SSE chunk (X-Accel-Buffering: no)
    Note over E: Must NOT buffer
    E-->>B: SSE chunk
    Note over B: EventSource fires onmessage
    Note over B: JS processes delta: ~1ms
    Note over B: DOM update

The full TTFT budget from the user's perspective:

Provider prefill:      300-600ms  (varies by model, context length)
Provider first chunk:  +50ms      (internal provider scheduling)
App server relay:      +5-15ms    (decode SSE chunk, re-encode, write)
CDN/edge passthrough:  +10-30ms   (with correct no-buffer config)
Network RTT:           +20-100ms  (varies by geography)
Browser/JS processing: +1-5ms

Total TTFT range: ~400ms - 750ms with everything working correctly
With CDN buffering:    +4-8s      (CDN waits for full response)

If your TTFT is in the 4-8 second range despite streaming being enabled, the first thing to check is CDN and proxy buffering headers, not the provider or your application code.

When to use the batch API instead

Streaming makes sense when a human is waiting and reading. It is the wrong choice for:

• Bulk document processing where completions feed a database or queue
• Classification pipelines where the output is a label, not text to read
• Embedding generation (no streaming API exists)
• Background summarization that runs overnight

For these workloads, the batch API from OpenAI and Anthropic cuts cost by ~50% and removes latency SLO pressure entirely. The mental model: streaming optimizes for time-to-first-token, batch API optimizes for cost and throughput. Pick based on whether a human is waiting. Mixing both in the same codebase — streaming for interactive features, batch for data pipelines — is the correct production pattern for most applications.

The judgment call in practice

Streaming is the right default for any user-facing LLM integration. The engineering cost is low: switch stream: false to stream: true, wire req.on("close") → AbortController, add a debounce on markdown rendering. Most of the complexity — partial-JSON parsing for tool calls, backpressure in relay servers — only appears when you already have real production traffic and real tool-using agents.

The order in which to implement these:

1. Enable streaming with TTFT monitoring from day one. Perceived latency is part of your product quality.
2. Wire client-disconnect cancellation before launch. This is a correctness fix, not an optimization.
3. Add backpressure handling to relay servers before load testing. Discover it in staging, not production.
4. Implement partial-JSON buffering for tool calls when you add tool use — required, not optional.
5. Tune markdown rendering debouncing once you have user feedback on flicker.

The most common mistake is treating streaming as a drop-in flag change and shipping it without cancellation. The second most common is implementing cancellation only on the client side, without the server-side relay abort that actually stops the provider from generating. Do both. Everything else is tuning.

For the broader latency budget picture — how streaming fits alongside model routing, caching, and serving configuration — see the latency and throughput tradeoffs article. For what happens after your gateway receives the stream — observability, cost attribution, and rate limiting across providers — see the LLM gateway pattern.