The Voice Pipeline: STT to LLM to TTS End-to-End

Your voice agent demoed beautifully on localhost. The STT returned in 120ms, the LLM fired back in 200ms, the TTS started playing in 50ms. Then you shipped it to real users and your monitoring showed an average TTFA of 1,100ms. Nobody changed a line of code. What happened?

Three things happened: your laptop is in the same AWS region as your inference endpoints, your demo audio was clean studio speech, and you measured the happy path. Production delivers 4G callers in Mumbai talking to a US-east inference cluster through overlapping background noise, and your pipeline is running each stage sequentially, waiting for a full transcript before starting the LLM and waiting for a complete LLM response before starting TTS. At that point the math is unforgiving: 300ms + 500ms + 120ms = 920ms before a user hears anything. And that's on a good day.

This article walks the full pipeline — every component, every handoff, and the specific techniques that compound to close that gap.

The three stages and what each one does

The term "voice pipeline" covers three technically distinct systems that happen to be chained together.

Speech-to-text (STT/ASR) converts raw audio bytes into text. The audio arrives as a PCM stream — typically 16 kHz, 16-bit mono for voice agents — and the model produces either a final transcript (batch mode) or a rolling stream of partial and final words (streaming mode). For real-time agents, streaming mode is non-negotiable. The only useful metric is time-to-final-word on the spoken utterance, measured from when the user stops talking. The leading providers as of mid-2026:

• Deepgram Nova-3: ~150ms p50 streaming latency, best word error rate for English call-center audio
• AssemblyAI Universal-2: higher WER on clean speech but consistently better on accented and overlapping audio
• OpenAI Whisper API: designed for batch transcription; its internal buffering adds 500ms–1s of overhead that makes it unsuitable for sub-300ms streaming pipelines regardless of how fast the underlying model is

LLM inference takes the text transcript and generates a response. The critical metric here is TTFT (time-to-first-token), not total generation time. Voice agents typically generate 50–200 tokens per turn, but TTFT is what determines when TTS can start. TTFT is also the most variable component: it depends on prompt length, the provider's current load, whether a prefix cache is warm, and the geographic distance between your inference call and the provider's servers.

Text-to-speech (TTS) converts the model's text output into audio. The metric is TTFB (time-to-first-audio-byte) — how long from the first text input token until the first audio chunk is ready to play. Modern streaming TTS engines accept tokens as they arrive from the LLM, accumulate enough phonemes to synthesize a short chunk, and start streaming audio without waiting for the full sentence. The current best:

• Cartesia Sonic 4: ~40ms TTFB, the lowest latency production TTS available as of mid-2026
• ElevenLabs Flash v2.5: ~75ms TTFB, stronger voice expressiveness than Cartesia
• Deepgram Aura-2: ~90ms TTFB, optimized for telephony audio profiles

The streaming architecture

Understanding why the naive pipeline is slow makes the streaming fix obvious.

sequenceDiagram
    participant U as User
    participant STT as STT
    participant LLM as LLM
    participant TTS as TTS
    participant SPK as Speaker

    Note over U,SPK: Sequential (no streaming) — TTFA ~900ms (from speech end)
    U->>STT: speaks, stops at t=300ms
    STT-->>LLM: final transcript (t=600ms, 300ms STT)
    LLM-->>TTS: full response (t=1,100ms, 500ms LLM)
    TTS-->>SPK: first audio chunk (t=1,200ms, 100ms TTS)

Each stage waits for the previous stage to finish completely before starting. The TTFA is roughly the sum of all three stage latencies.

sequenceDiagram
    participant U as User
    participant STT as STT
    participant LLM as LLM
    participant TTS as TTS
    participant SPK as Speaker

    Note over U,SPK: Streaming — TTFA ~340ms (from speech end)
    U->>STT: starts speaking
    STT-->>LLM: partial transcript at t=80ms
    STT-->>LLM: partial transcript at t=160ms
    U->>STT: stops speaking at t=300ms
    STT-->>LLM: final transcript (t=450ms, 150ms STT)
    LLM-->>TTS: first tokens (t=600ms, 150ms cached TTFT)
    TTS-->>SPK: first audio chunk (t=640ms, 40ms TTS)
    LLM-->>TTS: more tokens (ongoing)
    TTS-->>SPK: more audio chunks (ongoing)

With streaming, STT sends partial transcripts to the LLM as words land. The LLM can start processing context and generating a response before the user has finished speaking — shaving 100–200ms. The LLM streams tokens to TTS, which accumulates a sentence boundary (roughly 8–12 words) and fires the first audio chunk without waiting for the full response. TTS playback begins while the LLM is still generating the rest of the answer.

The sentence-boundary flush is worth understanding in detail. TTS engines perform best when they have complete phoneme context — a full sentence gives the model better prosody than a fragment. But waiting for the full LLM response to flush TTS adds 400–800ms. The practical compromise: flush TTS at the first sentence boundary (., !, ?, or a clause-ending comma followed by enough words). This gives TTS enough context for natural prosody on the first sentence while the LLM generates the rest of the response in parallel.

{ "type": "voice-latency", "title": "Voice pipeline: sequential vs streaming" }

Latency budget worked example

Here is a concrete budget for a customer support agent handling English-language calls:

Pipeline components (streaming, mid-2026 providers):
──────────────────────────────────────────────────────────────
Component         Provider               p50 latency  p95 latency
─────────────────────────────────────────────────────────────
STT (streaming)   Deepgram Nova-3        150ms        280ms
LLM TTFT          GPT-4o (cached pfx)   180ms        400ms
TTS TTFB          Cartesia Sonic 4       40ms         90ms
Network (total)   US-East ↔ US-East      30ms         80ms
─────────────────────────────────────────────────────────────
TTFA (streaming)  Sum of above           400ms        850ms

Without prefix caching on LLM:           +200ms       +300ms
Without streaming (sequential):          +400ms       +600ms
On a cold LLM instance (no GPU warmup):  +300ms       +800ms
─────────────────────────────────────────────────────────────
Production average without optimization: ~1,000ms     ~1,700ms
(no cache + no streaming; cold instances hit the tail, not the average)

The gap between 400ms and 1,000ms is almost entirely two decisions: whether you stream partials between stages, and whether your LLM system prompt prefix is cached. Both are free to implement once you understand them. This is why most voice teams who complain about latency haven't yet done the cheapest optimizations.

STT in depth

Streaming STT produces two types of events: partial (in-flight, may change) and final (stable, word is committed). The right strategy is to send partials to the LLM for context but start LLM generation only on final words, to avoid wasting prefill computation on transcripts that will be corrected.

Deepgram's streaming API emits final words with is_final: true within roughly 100–200ms of the word being spoken — not after the sentence is done. The snippet below still triggers the LLM only at utterance end (speech_final) — the latency win here is that the words are already committed by then, so the final transcript is ready ~150ms after speech end instead of 300ms+. True partial-transcript overlap (speculative prefill on in-flight words) is a further optimization on top of this:

import asyncio
from deepgram import AsyncListenWebSocketClient, LiveOptions, LiveTranscriptionEvents
from anthropic import AsyncAnthropic

anthropic = AsyncAnthropic()

async def stream_voice_turn(audio_source, tts_ws):
    dg = AsyncListenWebSocketClient()
    transcript_buffer = []

    async def on_transcript(result, **kwargs):
        if result.is_final:
            words = result.channel.alternatives[0].transcript
            transcript_buffer.append(words)
            # Start LLM once we have a complete utterance
            if result.speech_final:
                full_transcript = " ".join(transcript_buffer)
                await generate_and_speak(full_transcript, tts_ws)
                transcript_buffer.clear()

    dg.on(LiveTranscriptionEvents.Transcript, on_transcript)

    opts = LiveOptions(
        model="nova-3",
        language="en",
        smart_format=True,
        utterance_end_ms="1000",
        vad_events=True,
    )
    await dg.start(opts)
    # feed audio_source chunks to dg here

async def generate_and_speak(transcript: str, tts_ws):
    # tts_ws: open TTS WebSocket (see the TTS section below)
    async with anthropic.messages.stream(
        model="claude-sonnet-4-5",
        max_tokens=300,
        system=[{
            "type": "text",
            "text": SYSTEM_PROMPT,
            # caching is opt-in — without this marker the prefix
            # is reprocessed cold on every turn
            "cache_control": {"type": "ephemeral"},
        }],
        messages=[{"role": "user", "content": transcript}],
    ) as stream:
        sentence_buf = ""
        async for text in stream.text_stream:
            sentence_buf += text
            # Flush to TTS at sentence boundaries
            if any(sentence_buf.endswith(c) for c in [".", "!", "?"]):
                await send_to_tts(sentence_buf.strip(), tts_ws)
                sentence_buf = ""
        if sentence_buf.strip():
            await send_to_tts(sentence_buf.strip(), tts_ws)

The speech_final flag is Deepgram's signal that the utterance is complete — distinct from is_final on individual words. That's the trigger to send the full transcript to the LLM.

LLM TTFT: the variable everyone ignores

STT latency is measurable and relatively stable. TTS latency is well-specified by providers. LLM TTFT is the one that swings wildly and is least well-understood by teams new to voice.

Prompt prefix caching is the highest-ROI optimization available. Every major provider supports it: OpenAI's cached input tokens at 50% discount, Anthropic's prompt caching at 90% discount on cache hits, and Google's context caching for Gemini. In practice: move your system prompt, persona instructions, tool definitions, and any static context to the top of the prompt and keep it stable. The provider caches the KV state for those tokens. A 2,000-token system prompt costs the same to cache once as to process cold; subsequent calls skip the prefill computation for those tokens, cutting TTFT by 200–400ms.

# Anthropic — explicit cache_control markers
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": transcript,
            }
        ],
    }
]

system = [
    {
        "type": "text",
        "text": SYSTEM_PROMPT,  # 2,000+ tokens of stable instructions
        "cache_control": {"type": "ephemeral"},  # cache this prefix
    }
]

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=300,
    system=system,
    messages=messages,
)

Regional routing is the second lever. LLM providers run inference in a handful of regions; the network round-trip from your compute to their GPU cluster is a hard floor. If your users are in Europe and you are calling api.openai.com hitting US-East, you are adding 80–150ms of pure transit latency. Route to the nearest inference endpoint. On OpenAI this means the us vs eu vs ap API base URLs; on Anthropic, regional endpoints via AWS Bedrock or GCP Vertex with specific region selection.

TTS in depth

TTS systems designed for real-time voice agents operate in streaming mode: they accept a text stream and output audio chunks without waiting for the full input. Cartesia's Sonic 4 API accepts WebSocket streaming input:

import asyncio
import cartesia

async def send_to_tts(text_chunk: str, ws_connection):
    await ws_connection.send({
        "context_id": "turn_1",
        "model_id": "sonic-4",
        "transcript": text_chunk,
        "voice": {"mode": "id", "id": VOICE_ID},
        "output_format": {
            "container": "raw",
            "encoding": "pcm_f32le",
            "sample_rate": 44100,
        },
        "continue": True,  # more text is coming
    })
    # response is a stream of audio chunks over the same WebSocket

Setting continue: True tells Cartesia that more text is coming on the same context — it will not finalize the prosody or add end-of-sentence silence until you send continue: False. This lets you stream the LLM response sentence by sentence without the TTS treating each sentence as an isolated utterance.

The choice between TTS providers comes down to three variables: latency, voice quality, and cost:

Provider	TTFB	Voice quality	Cost (illustrative)
Cartesia Sonic 4	~40ms	Good, less expressive	~$0.025/1K chars
ElevenLabs Flash v2.5	~75ms	Excellent, most expressive	~$0.11/1K chars
Deepgram Aura-2	~90ms	Neutral, telephony-tuned	~$0.015/1K chars
OpenAI TTS-1-HD	~200ms	Very good	~$0.030/1K chars

At 100,000 minutes per month of calls averaging 120 characters of TTS output per turn and 10 turns per minute, that's 120M characters/month. The cost gap between Cartesia ($3,000/month) and ElevenLabs ($13,200/month) is real money. The 35ms latency difference between them is imperceptible on most calls. Pick Cartesia for cost-sensitive pipelines; ElevenLabs Flash v2.5 when voice expressiveness is a product differentiator.

VAD and turn detection

Before any of the above, there is a problem the pipeline diagram glosses over: how does the pipeline know the user has stopped speaking?

VAD (voice activity detection) classifies audio frames as speech or silence. Most production pipelines use one of two approaches. Energy-based VAD fires when audio energy drops below a threshold for N milliseconds — fast, cheap, but it also fires on mid-sentence pauses, filler words ("um, uh"), and brief hesitations. The effect in production: the agent interrupts users in mid-thought, which is the single most-cited user complaint in voice AI deployments.

Semantic VAD, available in the OpenAI Realtime API as of late 2025, runs a small model that waits for the utterance to be semantically complete before signaling turn-end. It adds ~100ms of latency but dramatically reduces false-positive interruptions. For most agent applications, that 100ms trade is worth making.

For a deep treatment of turn detection and barge-in handling, the VAD and turn detection article covers it in full — including the specific OpenAI Realtime API event sequence for interruption handling.

What breaks

Echo cancellation failures

When the user is on speakerphone, the agent's audio output leaks back into the user's microphone. Without AEC (acoustic echo cancellation), the STT picks up the agent's own voice and transcribes it. The LLM then receives that as user input, generates a response to its own words, and TTS plays that — which the STT picks up again. The pipeline enters a feedback loop and will eventually say something baffling. Twilio's server-side AEC handles typical cases, but it fails on cheap speakerphones and reverberant rooms. This is a hardware and room-acoustics problem as much as a software one; the reliable fix is ensuring AEC runs client-side before audio even reaches your server.

LLM TTFT spikes

LLM inference has tail latency problems that STT and TTS do not. A cold GPU instance, provider load spikes, or a long context length can push TTFT from 200ms to 1,200ms without warning. At p99 on a frontier model at peak hours, 1,500ms TTFT is not unusual. That alone blows your sub-500ms TTFA budget even with perfect streaming everywhere else. The mitigations: keep contexts short (voice turns are usually 50–100 tokens, not 2,000), use a smaller model for low-complexity turns (a 7B model with 120ms TTFT sometimes beats a frontier model with 400ms TTFT for simple Q&A), and have a fallback provider in your routing logic. The latency budget engineering article has worked numbers on each mitigation.

Tool calls blow every budget in this article

The budgets above assume the LLM can answer from context. A customer support agent usually cannot — the turn that matters is "where's my order?", which means a CRM lookup or an order-API call mid-turn. A tool call adds a full extra cycle: the model emits the call (~200–400ms of generation), your backend hits the API (200ms for a fast internal service, 1–10s for a real CRM), and then the model runs a second inference over the result before any speakable text exists. TTFA on that turn is 1.5s at best and 10s+ at the ugly end. No amount of streaming discipline in the STT → LLM → TTS handoffs recovers it, because the wait is upstream of the first token.

The mitigation is not making the CRM faster — you usually can't — it is filling the silence with acknowledgement audio. Detect the tool-call event in the LLM stream and immediately flush a short filler to TTS: "Let me pull that up." The user hears a response inside the 500ms band while the lookup runs, and perceived latency stays acceptable even when actual latency is 4s. A prompt-level variant works too: instruct the model to speak one acknowledgement sentence before emitting the tool call, and the normal sentence-boundary flush handles the rest. Either way, track tool-call turns as their own TTFA histogram — averaging them into ordinary turns hides exactly the tail that pages you.

Partial transcript instability

Streaming STT partials can change significantly as more audio arrives. "I need to cancel" can temporarily parse as "I need to cans", and if you send that to the LLM immediately you waste prefill computation on garbage. The right approach is to wait for is_final word events from Deepgram — which come in at ~100–200ms per word — rather than streaming raw partials into the LLM.

TTS audio queue buildup

If the LLM generates tokens faster than TTS can synthesize and the network can deliver audio (common on fast-inference models), audio chunks queue client-side. When the user interrupts (barge-in), your pipeline needs to not only cancel the LLM call and discard buffered TTS but also clear the client-side audio queue. A user who barged in 400ms ago should not hear 600ms of stale audio from the interrupted response before the new response starts. Audio queue management is an implementation detail that most tutorials skip and every production deployment has to solve.

PSTN audio quality degradation

Public telephone networks deliver 8 kHz audio with codec artifacts that significantly degrade STT accuracy. Deepgram's Aura-2 and AssemblyAI Universal-2 both have telephony-tuned variants that expect this audio profile. Using a model tuned for 16 kHz studio audio on 8 kHz PSTN input can push WER from 7% to 25%+. Always check that your STT model variant matches your audio source.

What to actually measure

Teams new to voice pipelines often instrument end-to-end latency and wonder why their optimization efforts don't feel like improvements. The right instrumentation:

import time

class VoiceTurnMetrics:
    def __init__(self):
        self.user_stopped_speaking_at = None
        self.stt_final_at = None
        self.llm_first_token_at = None
        self.tts_first_audio_at = None
        self.llm_last_token_at = None

    @property
    def ttfa(self):
        """Time to first audio — the metric that predicts UX quality."""
        if self.user_stopped_speaking_at and self.tts_first_audio_at:
            return self.tts_first_audio_at - self.user_stopped_speaking_at
        return None

    @property
    def stt_latency(self):
        if self.user_stopped_speaking_at and self.stt_final_at:
            return self.stt_final_at - self.user_stopped_speaking_at
        return None

    @property
    def llm_ttft(self):
        if self.stt_final_at and self.llm_first_token_at:
            return self.llm_first_token_at - self.stt_final_at
        return None

    @property
    def tts_ttfb(self):
        if self.llm_first_token_at and self.tts_first_audio_at:
            return self.tts_first_audio_at - self.llm_first_token_at
        return None

Track ttfa, llm_ttft, and stt_latency as separate histograms. The first tells you if the user experience is good. The second and third tell you where to focus.

Cascaded pipeline vs native speech-to-speech

The architecture above is a cascaded pipeline — audio in, text intermediate, audio out, with separate models at each stage. The alternative is native speech-to-speech (S2S): models like OpenAI gpt-realtime-1.5 and Gemini 3.1 Flash Live that take audio in and produce audio out directly, skipping the text representation entirely.

The case for native S2S is real: emotional prosody (tone of voice, pacing) is preserved through the full model rather than being recreated synthetically by a TTS engine, and the audio-in to audio-out path removes the STT and TTS stages as bottlenecks. OpenAI gpt-realtime-1.5 benchmarks at ~820ms TTFA as of mid-2026 — which is not faster than an optimized cascaded pipeline, but the conversational voice quality is more natural.

flowchart LR
    subgraph Cascaded
    A1["Audio in"] --> B1["STT"] --> C1["Text"] --> D1["LLM"] --> E1["Text"] --> F1["TTS"] --> G1["Audio out"]
    end
    subgraph "Native S2S"
    A2["Audio in"] --> B2["Single model"] --> C2["Audio out"]
    end
    style B1 fill:#0e7490,color:#fff
    style D1 fill:#a855f7,color:#fff
    style F1 fill:#15803d,color:#fff
    style B2 fill:#00e5ff,color:#0a0a0f

The case against native S2S for most production deployments is cost and debuggability. OpenAI gpt-realtime-1.5 costs ~$0.30/min versus ~$0.02–$0.04/min for a cascaded pipeline at the same quality level. And without a text intermediate, you have no transcript for compliance logging, CRM writes, or post-call analytics — you need a separate transcription pass to get that, which partly defeats the purpose.

The decision matrix is covered in depth in the cascaded vs native S2S article. The short version: native S2S when emotional coherence and voice quality are primary; cascaded when cost, debuggability, telephony compatibility, or tool-call reliability matter more.

The pipeline inside a voice agent framework

Building the streaming handoffs described here from scratch takes two to four weeks. Most production teams use a framework that handles it:

• Vapi: fully managed, pay-per-minute, built-in telephony. Handles all the streaming plumbing internally. Right for < 10K min/month or teams without dedicated voice engineering.
• LiveKit Agents 1.5.x: open-source WebRTC infrastructure with a Python SDK. You assemble the pipeline from STT/LLM/TTS components; LiveKit handles audio transport, echo cancellation, and the WebRTC media layer.
• Pipecat v1.0: open-source Python framework from Daily.co. Explicit pipeline definition, maximum control, you own latency tuning.

The voice agent frameworks article covers the decision matrix.

The judgment call

The voice pipeline is not a solved problem — it is a set of well-understood tradeoffs that you assemble to hit a specific latency and cost target.

For a production customer support agent in 2026: Deepgram Nova-3 for STT, a frontier model (GPT-4o or Claude Sonnet) with prefix caching for LLM, and Cartesia Sonic 4 for TTS. Run on LiveKit Agents if you have the engineering bandwidth, Vapi if you do not. Measure TTFA, not end-to-end time. Implement streaming at every handoff before touching anything else — it is the single highest-leverage change, and most teams ship without it.

The 800ms industry average is not a floor. It is what you get when you take three good components and connect them naively. The ceiling is around 340ms with current providers, achieved by overlapping every stage and caching your LLM prefix. The difference between those two numbers is your engineering problem, and it is entirely solvable.

For latency optimization beyond what this article covers — regional routing math, TTS sentence-boundary flushing strategies, and the specific OpenAI Realtime API optimizations — see latency budget engineering. For the architectural question of whether to use a cascaded pipeline at all, see cascaded vs native speech-to-speech.