Voice AI latency budget: hitting sub-500ms in production

A voice agent demo shipped at an internal all-hands. The STT vendor slide said 150ms. The LLM vendor slide said 200ms TTFT. The TTS vendor slide said 40ms. Someone did the math on a whiteboard: 390ms. Everyone nodded. Six weeks later the agents were in production and user research came back with one consistent word: "laggy." The p50 TTFA was 940ms. The p95 was 2.1 seconds.

Nothing lied. The components genuinely hit those numbers — in isolation, on clean studio audio, against a warmed cache, in the same AWS region as the test machine. None of those conditions held in production. This article is about the gap between component specs and production voice AI latency, and the systematic techniques that close it.

The voice pipeline budget: where every millisecond lives

The voice pipeline has four latency-producing stages. Before optimizing any of them, measure them all separately with real production traffic, because the interventions are different for each.

Stage 1: STT (speech-to-text). The clock starts when voice activity detection fires the end-of-turn event — the moment VAD decides the user has stopped speaking. (If VAD is misconfigured, this fires 200–400ms too late; that is a turn detection problem, not a STT problem.) From that moment, STT must deliver a final transcript. Deepgram Nova-3 achieves 100–150ms on typical utterances in the same region. AssemblyAI Universal-2 runs 150–250ms but with better accuracy on accented or noisy speech. Whisper — including the OpenAI-hosted version — is not a streaming-grade model; batch Whisper adds 500ms–2s and should not be in a real-time voice pipeline.

Stage 2: LLM TTFT. The time-to-first-token is the single highest-variance stage. A small cached request to a fast model in the same region can deliver tokens in 150ms. An uncached, large-context request to a frontier model in the wrong region takes 800–1,200ms. The range is six-to-one. This is also the most improvable stage — the techniques in the next section attack it from multiple angles.

Stage 3: TTS TTFB. Time-to-first-byte from the TTS provider. Cartesia Sonic 4 is ~40ms as of mid-2026 — the lowest in production. ElevenLabs Flash v2.5 is ~75ms. Deepgram Aura-2 is ~90ms. These numbers are from independent benchmarks on real workloads; vendor-quoted numbers are tested at zero concurrency on clean text. At high concurrency, expect 30–50% degradation. The TTS stage also has an underappreciated internal coupling point: if you send the full LLM response to TTS as a single request, TTS TTFB does not start the clock until the LLM response is complete. That makes TTS latency effectively LLM_total_generation_time + TTS_TTFB, not just TTS_TTFB.

Stage 4: Network round-trips. Every API call is a round-trip. An audio chunk upload to STT, an LLM API call, a TTS API call. On same-region cloud infrastructure each round-trip is 10–30ms. Cross-region, US-East to EU-West, adds 80–120ms per hop. A three-stage pipeline with two network hops in the wrong region adds 160–240ms before any model has done any work.

sequenceDiagram
    participant U as User utterance
    participant VAD as VAD
    participant STT as STT API
    participant LLM as LLM API
    participant TTS as TTS API
    participant SPK as Speaker

    U->>VAD: audio stream
    VAD->>STT: end_of_turn + audio (t=0)
    STT-->>LLM: final transcript (t=150ms)
    LLM-->>TTS: first tokens (t=350ms)
    TTS-->>SPK: first audio bytes (t=390ms)
    Note over U,SPK: Sequential: TTFA ~390ms best case

Sequential coupling is the core problem. Each stage waits for the previous stage to complete. The components then add up rather than overlap. The optimizations below all attack the same structural issue: replace sequential waits with parallel streaming.

The highest-ROI technique: stream across stage boundaries

The single biggest latency improvement — available to almost every team regardless of infrastructure — is streaming STT partials directly into the LLM while the STT transcription is still running.

Deepgram and AssemblyAI both emit partial transcripts in real time, typically within 50–100ms of a phrase being spoken. The partial transcript is not final — it may be revised — but it is usually directionally correct. Instead of waiting for the is_final: true event, you can start an LLM inference call with the partial transcript and either:

1. Cancel and restart if the partial is revised significantly before the final arrives.
2. Let the LLM infer on the partial and handle partial-transcript noise in the system prompt.

Option 2 is simpler and works well enough in practice because partials are revised only modestly on the last few words. The saving is the time the LLM would otherwise spend idle waiting for the final transcript: typically 200–400ms depending on utterance length and STT provider.

import asyncio
from deepgram import DeepgramClient, LiveOptions, LiveTranscriptionEvents
from openai import AsyncOpenAI

oai = AsyncOpenAI()
dg = DeepgramClient()

MIN_PARTIAL_WORDS = 4  # don't speculate on "so I was"

async def handle_transcription(call_id: str, audio_stream):
    """Stream STT partials into LLM without waiting for final transcript."""
    llm_task = None
    speculative_text = ""

    # Deepgram delivers both partials and finals as Transcript events;
    # branch on result.is_final. (UtteranceEnd carries no transcript.)
    async def on_transcript(_conn, result, **kwargs):
        nonlocal llm_task, speculative_text
        text = result.channel.alternatives[0].transcript
        if not text:
            return

        if not result.is_final:
            # Fire the speculative LLM call ONCE, on the first partial long
            # enough to be worth inferring on. Partials arrive every 50–100ms;
            # cancel-and-restart on each one burns tokens on aborted calls,
            # and only the last restart would determine the saving anyway.
            if llm_task is None and len(text.split()) >= MIN_PARTIAL_WORDS:
                speculative_text = text
                llm_task = asyncio.create_task(stream_llm_response(text))
            return

        # Final transcript: keep the speculative call if it was close enough,
        # restart only on significant divergence.
        if llm_task is None:
            llm_task = asyncio.create_task(stream_llm_response(text))
        elif not partial_close_enough(speculative_text, text):
            llm_task.cancel()
            llm_task = asyncio.create_task(stream_llm_response(text))

    connection = dg.listen.asynclive.v("1")  # async client for async callbacks
    connection.on(LiveTranscriptionEvents.Transcript, on_transcript)
    await connection.start(LiveOptions(
        model="nova-3",
        interim_results=True,   # emit partials, not just finals
        smart_format=True,
    ))

    async for chunk in audio_stream:  # pump audio; start() takes no audio
        await connection.send(chunk)
    await connection.finish()


async def stream_llm_response(user_text: str):
    """Stream tokens from LLM and flush to TTS at sentence boundaries."""
    stream = await oai.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_text},
        ],
        stream=True,
        max_tokens=300,
    )

    buffer = ""
    async for chunk in stream:
        delta = chunk.choices[0].delta.content or ""
        buffer += delta
        # Flush TTS at sentence boundaries — the key technique
        if any(buffer.endswith(p) for p in (".", "!", "?", ":", "\n")):
            asyncio.create_task(send_to_tts(buffer.strip()))
            buffer = ""

    if buffer.strip():
        asyncio.create_task(send_to_tts(buffer.strip()))

The pattern to observe: stream_llm_response does not wait for the full LLM response before calling TTS. It flushes whenever it hits a sentence-ending punctuation mark. This is the second highest-ROI technique.

TTS sentence-boundary flushing

When the LLM emits a complete sentence, that sentence is self-contained — the TTS can render and play it while the LLM is still generating the next sentence. The user hears the first sentence begin at LLM_TTFT + TTS_TTFB (roughly 190–375ms on an optimized stack), not at LLM_total_generation_time + TTS_TTFB.

The saving is proportional to how long the full LLM response would have been. A 3-sentence response where each sentence takes 200ms to generate means the user would otherwise wait 600ms + 40ms = 640ms. With sentence-boundary flushing, they hear the first sentence at TTFT + 40ms, say 240ms total. That is a 400ms improvement on a medium-length response.

The failure mode is flushing too eagerly. Flushing at every comma produces fragmented audio that sounds robotic. Flushing at . ! ? is the standard heuristic; some systems also flush at : before a list, or at \n before a paragraph. Do not flush mid-clause.

sequenceDiagram
    participant LLM as LLM token stream
    participant BUF as Sentence buffer
    participant TTS as TTS API
    participant SPK as Speaker

    LLM->>BUF: "Hello, I can help"
    LLM->>BUF: " with your account."
    BUF->>TTS: flush sentence 1 (at ".")
    LLM->>BUF: " Your balance is"
    TTS-->>SPK: playing sentence 1
    LLM->>BUF: " $240."
    BUF->>TTS: flush sentence 2 (at ".")
    Note over LLM,SPK: Overlap: user hears sentence 1 while LLM generates sentence 2

LLM prefix caching

The LLM's prefill phase — processing the input tokens — dominates TTFT on every request. For a voice agent, a large fraction of that input is static: the system prompt, tool definitions, persona instructions, and maybe the first few turns of a standard greeting. This static portion can be cached by the provider and skipped on subsequent requests.

Anthropic and OpenAI both support prompt caching. On Anthropic, cache breakpoints are explicit; on OpenAI, the API automatically caches the longest stable prefix. The saving is significant: on a 2,000-token system prompt at gpt-4o-mini, a cache hit saves roughly 200–400ms TTFT compared to a cold start.

The thing that silently kills this optimization is cache busting. Common culprits:

• Timestamp in the system prompt. "Today is {{datetime.now()}}" — busted on every request.
• User ID injected at the start of the system prompt. Move user-specific context to the end of the prompt, after the stable prefix.
• Shuffled tool definitions. If your code generates tool JSON from a dictionary with non-deterministic ordering, the prefix changes every call.
• Conversation history inserted into the system prompt. History should go in the user/assistant message turns, not in the system prompt prefix.

The fix is simple: audit what is in your system prompt, move dynamic content to the end or into user message turns, and verify caching is working by checking the cached_tokens field in the API response.

# Check that prefix caching is actually working
response = await oai.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    stream=False,  # easier to inspect for debugging
)
usage = response.usage
cached = getattr(usage, "prompt_tokens_details", None)
if cached:
    print(f"Cached tokens: {cached.cached_tokens}/{usage.prompt_tokens}")
    # If cached_tokens is 0 on the second call, something is busting the cache

Geographic routing

This is the lowest-effort optimization with a guaranteed return: deploy your STT, LLM, and TTS calls to the region closest to your users.

Network RTT from US-East to EU-West is 80–120ms. A two-hop pipeline (audio to STT, text to LLM) already adds 160–240ms of pure geography. For a pipeline targeting 500ms TTFA, that is one-third of the entire budget gone before a model processes anything.

The mechanism is straightforward: use providers that offer regional endpoints (Deepgram has US/EU/AU; OpenAI has us-east, eu, apac; Cartesia has regional inference), and route each API call to the closest region based on the caller's IP. For Kubernetes deployments, this means running the voice orchestration layer in multiple regions with geolocation-based load balancing at the edge.

One wrinkle: LLM model availability varies by region. If your preferred model is only available in one region, you are stuck with the RTT tax. Check regional availability before committing to a model.

The interactive budget: watching the numbers move

{ "type": "voice-latency", "title": "Voice pipeline latency budget" }

The visualizer makes the streaming-vs-sequential gap concrete. Toggle the streaming mode and watch the swimlanes overlap. The time-to-first-audio badge changes color at the 500ms threshold.

Cascaded pipeline vs native speech-to-speech

The obvious question: can you skip all of this by using a native speech-to-speech model that processes audio in and produces audio out without a text intermediate?

The answer is nuanced. Native S2S models like OpenAI gpt-realtime-1.5, xAI Grok Voice, and Gemini 3.1 Flash Live do eliminate the STT and TTS component latencies — but they introduce their own inference floor. Independent benchmarks in April 2026 show:

Model	TTFA (ms)	Cost/min	Transcript
OpenAI gpt-realtime-1.5	~820	~$0.30	No stable intermediate
Gemini 3.1 Flash Live	~2,980	~$0.04	No stable intermediate
xAI Grok Voice	~780	Not public	No stable intermediate
Amazon Nova Sonic	~1,140	~$0.02	No stable intermediate
Cascaded (Deepgram + gpt-4o-mini + Cartesia)	~350–450	~$0.05–0.08	Yes

The cascaded pipeline, when properly optimized with streaming and caching, beats every current native S2S model on raw TTFA. The reasons are worth understanding rather than just accepting:

1. Model size. Frontier S2S models are large to maintain audio fidelity and conversational quality. A cascaded pipeline can use a fast small LLM (like gpt-4o-mini) for most turns and escalate to a larger model only when needed.
2. Parallelism. A cascaded pipeline can stream the STT partial into an already-warm LLM while TTS is idle. A native S2S model processes the input audio serially before generating any output audio.
3. No intermediate transcript. The absence of a text intermediate is also the model's constraint — it cannot re-process or correct a transcript before speaking.

Native S2S wins on three things: emotional prosody (the voice carries tone from the user's original speech rather than inferring from text), simplicity (one API call instead of three coordinated streaming connections), and potentially interruption handling (audio-native models can handle barge-in more naturally). For the cascaded vs native decision at architecture time, those factors matter. For the latency budget, cascaded is the answer today.

What breaks

Energy-based VAD fires too early

Energy-based VAD fires when audio drops below a decibel threshold for a configured silence window. When a user pauses mid-sentence ("so I wanted to... um... ask about my subscription"), energy VAD fires after the silence window and sends a partial utterance to STT. The pipeline responds to an incomplete question, interrupts the user mid-thought, and the experience feels like a bad IVR menu from 2008.

Semantic VAD waits until the utterance is semantically complete — the model understands that "so I wanted to... um..." is not a finished thought. It costs ~100ms more than energy VAD but the user-experience improvement is dramatic. OpenAI Realtime API's Semantic VAD mode (available as of mid-2025) is the production default on any deployment where user experience matters.

The VAD and turn detection article covers this more deeply.

Echo cancellation failure → feedback spiral

If the agent's outgoing audio leaks into the user's microphone — which happens on speakerphone, cheap headsets, or any device without hardware AEC — the STT transcribes the agent's own speech. The LLM then generates a response to itself. The loop repeats.

This is not a latency problem but it is the most frequent production blocker teams discover after the "clean studio audio" phase. Hardware echo cancellation on WebRTC clients handles this correctly. Twilio's server-side AEC is insufficient for cheap speakerphones and poor acoustics — if your user base is on mobile speakerphone or in call centers, you need client-side AEC or a carrier-level solution.

Cache busting under load

LLM prefix caching seems to work in testing and then mysteriously provides no benefit in production. The most common cause: the conversation history grows as the call progresses, and if you concatenate history into the system prompt, the static prefix shrinks and eventually disappears. History must go in the messages array as user/assistant turns, not prepended to the system prompt.

Second cause: A/B testing or per-user system prompt customization that injects user attributes early in the prompt. Move per-user customization to a <context> block at the end of the system prompt, after the stable shared prefix that the cache can reuse across users.

Wrong latency metric in production dashboards

Teams that monitor "average response generation time" (LLM input to final token) rather than TTFA make optimization decisions against the wrong metric. A TTS provider change might cut TTFA by 50ms but show no improvement on the generation time dashboard. The latency metric in your observability stack must be: time from vad_end_of_turn to tts_first_byte_played. Two clarifications on that definition. First, the clock starts at vad_end_of_turn, not at the acoustic end of speech — the endpointing silence window (300–700ms for energy VAD) elapses before it, which is why a dashboard-perfect 430ms TTFA can still feel like a ~900ms wait to the user. Second, "played" means audible on the user's device, not received by your media server: the WebRTC jitter buffer adds 20–60ms, and a PSTN/telephony leg can add 100ms+ of carrier latency. Instrument the playout timestamp client-side where you can; server-side "first byte sent" quietly flatters the number.

TTS concurrency degradation

TTS TTFB numbers from vendor benchmarks are tested at near-zero concurrency. Under load — 50+ simultaneous calls — Cartesia Sonic 4's ~40ms becomes 60–100ms; ElevenLabs under load pushes into 150–200ms territory. If your latency degrades under traffic, TTS concurrency is often the culprit before STT or LLM. Load test your TTS provider at expected peak concurrency before committing.

flowchart TD
    START[Production voice call] --> VAD{VAD mode?}
    VAD -->|Energy-only| RISK1[Risk: fires on mid-sentence\npauses → interrupts user]
    VAD -->|Semantic| SAFE1[Safe: waits for semantic\ncompleteness]

    START --> CACHE{LLM prefix\ncached?}
    CACHE -->|Cold cache| RISK2[300–500ms TTFT tax\non first call]
    CACHE -->|Warm cache| SAFE2[150–200ms TTFT]

    START --> AEC{Echo\ncancellation?}
    AEC -->|Missing| RISK3[Feedback spiral:\nagent transcribes itself]
    AEC -->|Present| SAFE3[Clean transcription]

    style RISK1 fill:#ff2e88,color:#fff
    style RISK2 fill:#ffaa00,color:#0a0a0f
    style RISK3 fill:#ff2e88,color:#fff
    style SAFE1 fill:#15803d,color:#fff
    style SAFE2 fill:#15803d,color:#fff
    style SAFE3 fill:#15803d,color:#fff

Worked numbers: getting from 940ms to 430ms

The production pipeline that came in at 940ms p50 TTFA from the opening story had the following problems on diagnosis:

Before optimization
───────────────────
VAD fires                         t = 0
Audio upload to STT               +30ms  (network, same region)
STT processes, returns final      +280ms (no streaming partials)
LLM API call                      +25ms  (network)
LLM prefill + first token         +260ms (uncached — the timestamped
                                          3,000-token prompt busts the
                                          prefix cache on every call)
LLM first token arrives           +595ms from VAD
TTS call with full LLM response   +wait for all LLM tokens...
LLM streams ~40 output tokens     +225ms more (~180 tok/s)
TTS receives full response        +820ms from VAD
TTS TTFB                          +120ms (ElevenLabs Turbo under load)
First audio plays                 +940ms from VAD — the measured p50

After optimization — streaming partials, prefix caching, sentence-boundary flush, switched to Cartesia, same region:

After optimization (best-case trace)
────────────────────────────────────
VAD fires                          t = 0
Audio upload to STT (streaming)    +30ms
STT sends usable partial           +80ms  (partial covers the utterance)
LLM call starts on partial         +85ms  (5ms network)
LLM prefill on cached prefix       +135ms (prefix cached, 50ms prefill)
LLM first token                    +145ms from VAD
First sentence complete            +255ms (~20 tokens at ~180 tok/s)
Sentence boundary flush to TTS     +255ms from VAD
TTS TTFB (Cartesia, low concur.)   +300ms (~45ms)
First audio plays                  +300ms from VAD

Verified p50 in production: 430ms

The 130ms gap between the best-case trace and the production p50 is real and itemizable: longer utterances hold the flush back behind a longer first sentence, roughly one call in seven missed the prefix cache after deploys and reconnects, and Cartesia's TTFB drifted toward 60–80ms at peak concurrency. The trace shows the floor; p50 is the floor plus the tail of ordinary bad luck.

The interventions, roughly in order of implementation effort vs payoff:

1. Switch to streaming STT partials — biggest absolute saving, moderate implementation complexity.
2. Sentence-boundary TTS flush — significant saving, low complexity once streaming is already in place.
3. Fix prefix caching — remove timestamp from system prompt, move history to message array. Low effort, 200–400ms saving.
4. Switch TTS provider (ElevenLabs Turbo → Cartesia Sonic 4 for this use case) — low effort, 75ms saving.
5. Confirm regional routing — already same region in this case, no change needed.

Framework considerations

How much of this you implement manually depends on what layer you are building at. The voice agent frameworks article covers the full comparison, but from a latency perspective:

Vapi handles streaming and sentence-boundary flushing internally. You do not configure these manually; the tradeoff is that you cannot inspect or tune the internals. For teams under ~10K minutes/month who want sub-500ms without building the plumbing, Vapi's defaults are reasonable.

LiveKit Agents (1.5.x, April 2026) gives you full control over the pipeline with the AgentSession and Agent abstractions (the 0.x VoicePipelineAgent was retired at 1.0 — its name now only appears in migration guides). You can subscribe to STT transcription events and wire your own sentence flush logic into the pipeline nodes. Adaptive interruption handling ships as a default in 1.5.x.

Pipecat (v1.0, April 2026) is the most explicit — you wire the pipeline yourself. STTService → LLMService → TTSService with configurable intermediate processors. If you want to implement the partial-streaming pattern from the code sample above, Pipecat is where you do it. You own the latency numbers.

For teams above 50K minutes/month, the per-minute cost of managed platforms becomes the dominant constraint and self-hosting on LiveKit or Pipecat makes economic sense. Below that threshold, the engineering cost of owning latency tuning usually outweighs the savings.

The judgment call: where to spend your next optimization hour

Once you have streaming across all three stage boundaries and prefix caching working, the remaining gains are smaller and harder to win. Here is how to triage:

If your p50 TTFA is above 700ms: streaming is probably not wired end-to-end, or the LLM cache is cold on most requests. Fix those before anything else.

If your p50 is 500–700ms: check geographic routing first (free optimization, immediate return), then audit your system prompt for cache-busting patterns, then consider whether a smaller/faster LLM model on most turns (model routing) would buy headroom.

If your p50 is 350–500ms and you want lower: you are in diminishing-returns territory. Native S2S models are not faster than this today, so that is not the answer. Speculative execution (starting the LLM on a predicted next-turn based on conversation context) can shave another 50–100ms on predictable conversation flows. Custom hosting of STT and TTS components eliminates provider-side queueing at high concurrency. These are real-engineering projects with significant complexity.

Sub-350ms TTFA for a cascaded pipeline with current provider infrastructure is at the edge of what is achievable without co-located compute. Telnyx's architecture — ASR, LLM inference, and TTS running on carrier-owned co-located infrastructure — breaks the 200ms RTT barrier by removing network hops between stages entirely. That path requires a carrier partnership, not a software change.

The cost, latency, and model selection article covers the LLM side of this tradeoff in depth. The streaming LLM responses article is worth reading before implementing the sentence-buffer pattern — there are edge cases around unicode, partial emoji sequences, and token-boundary misalignments that will cost you debugging time if you build the buffer naively.