Cascaded pipeline vs native speech-to-speech: when to use each

The production voice agent you demoed last month sounded great. Then the legal team asked for call transcripts. Then the telephony integration broke. Then the billing team saw the API invoice.

Welcome to the real decision between cascaded and native speech-to-speech architecture — not as a capability question, but as an engineering and economics problem.

What the two architectures actually are

A cascaded voice pipeline is a chain of three specialized models. An STT model — Deepgram Nova-3, AssemblyAI Universal-2, or Whisper — converts the incoming audio stream to a text transcript. That transcript goes into an LLM (GPT-4o, Claude Sonnet, Gemini Flash, whatever fits your quality and cost budget). The LLM's text response goes to a TTS model — Cartesia Sonic 4, ElevenLabs Flash v2.5, Deepgram Aura-2 — which synthesizes audio and streams it back. Text is the connective tissue throughout.

A native speech-to-speech model collapses the three stages into one. Audio arrives, the model processes it through a unified multimodal transformer, and audio comes out. There is no text intermediate in the hot path. OpenAI's gpt-realtime (GA August 2025, updated to gpt-realtime-1.5 in May 2026) works this way. So does Gemini 3.1 Flash Live (launched March 2026), Amazon Nova Sonic, and Hume EVI 3.

The conceptual appeal of native S2S is obvious: fewer handoffs, no modality conversion losses, the model "hears" the user's tone rather than reading a transcript of it. But conceptual appeal and production performance are different things. Let's look at each axis where they diverge.

Latency: the one claim that deserves scrutiny

The most common assumption is that native S2S is faster. It can be — but the comparison depends heavily on how well-optimized your cascaded stack is.

Take an unoptimized cascaded pipeline: STT waits for the user to finish speaking before transcribing (full utterance wait), then the LLM processes the full transcript, then TTS waits for the complete LLM response before synthesizing. In that sequential, non-streaming mode, you're looking at:

Unoptimized sequential cascaded (worst case):
  End-of-turn detection (silence VAD):       ~500ms
  STT full-utterance processing:             ~300ms
  LLM TTFT (no prefix caching):              ~800ms
  LLM full response (~40 tok at 80 tok/s):   ~500ms
  TTS full synthesis before playback:        ~200ms
  Network round-trips:                       ~150ms
  Total TTFA:                                ~2,450ms

Against that baseline, any native S2S model looks fast. But that is not the right comparison.

A properly optimized cascaded pipeline streams at every stage. Deepgram streams partial transcripts as the user speaks. The LLM receives partial input and begins generating before the user finishes. Cartesia Sonic 4 starts synthesizing the first sentence of the LLM's response before the rest arrives. Each stage overlaps with the next. The voice pipeline end-to-end article covers the streaming mechanics in detail; the short version is that this overlap, plus an aggressively tuned endpointer, cuts the sequential total by roughly three quarters.

Optimized streaming cascaded (achievable):
  End-of-turn detection (tuned VAD):         ~250ms
  STT transcript finalization (streaming
  partials already fed the LLM):             ~50ms
  LLM TTFT with prefix caching:              ~200ms
  TTS first sentence playback:               ~40ms (Cartesia Sonic 4)
  Network (regional routing, WebRTC):        ~60ms
  Total TTFA:                                ~600ms

One line in that budget never appears in vendor marketing diagrams: end-of-turn detection. TTFA is measured from the moment the user stops speaking, and the pipeline has to detect that stop — a tuned silence threshold still costs ~200–300ms, and semantic endpointing adds ~100ms on top. No amount of streaming removes it. Budget for 500–700ms TTFA in practice, not the 400ms you get by summing only the model stages.

{ "type": "voice-latency", "title": "Sequential vs streaming cascaded — where the TTFA goes" }

Now look at what native S2S models actually deliver in independent benchmarks from April 2026:

Model	TTFA (benchmarked)	Cost/min	Notes
xAI Grok Voice	~780ms	Early access	Fastest native S2S benchmark
OpenAI gpt-realtime-1.5	~820ms	~$0.30	GA, most production usage
Amazon Nova Sonic	~1,140ms	~$0.02	Lowest cost native option
Gemini 3.1 Flash Live	~2,980ms	Varies	90+ languages, higher latency

A well-tuned cascaded stack at 500–700ms TTFA is comparable to or faster than every production-grade native S2S model in the current generation. The narrative that native S2S is "lower latency" is true only relative to a naive sequential cascaded pipeline. Relative to an optimized streaming pipeline, native S2S holds no latency advantage today.

This will change. The models will get faster. But as of mid-2026, "go native for latency" is not yet correct advice.

The transcript problem

Here is where the choice becomes structural. Text is the cascaded pipeline's native byproduct — you have it at every stage. It flows from STT into the LLM, and the LLM's output text feeds the TTS. You can:

• Log every turn for compliance or audit
• Write to CRM in real time based on transcript content
• Route to different tools based on detected intent in the transcript
• Stream partial transcripts to a supervisor UI during a live call
• Run a sentiment classifier on the text mid-turn

Native S2S has no intermediate text. The model produces audio. OpenAI's Realtime API does return transcripts as a side channel, but they arrive after the audio — you cannot intercept a partial transcript mid-utterance to make a routing decision before the response has started playing. If your use case needs to look at the text content during the conversation to do anything (tool calling, compliance flagging, handoff to a human agent), you are fighting the architecture.

This is a real constraint for enterprise deployments. If a voice agent is handling customer service calls that touch PCI-DSS data, you need transcripts. If an agent is writing call summaries to Salesforce, you need transcripts. If a healthcare bot needs to detect a user mentioning medication names for documentation, you need transcripts. The cascaded pipeline is not just easier for these use cases — it is the only option without adding a parallel transcription stream, which partially defeats the simplification that native S2S offers.

Tool calls and RAG

Language model tool calls require text. When a voice agent needs to look up an account balance, book an appointment, or query a knowledge base, it needs to emit a structured tool call and parse the result. In a cascaded pipeline, this is straightforward: the LLM is just an LLM, it can call tools exactly as it would in a text context, the tool result feeds back into the LLM's context, and the cycle continues.

sequenceDiagram
    participant U as User (audio)
    participant STT as STT
    participant LLM as LLM
    participant TOOL as Tool / API
    participant TTS as TTS
    participant UA as User (audio out)

    U->>STT: "What's my account balance?"
    STT->>LLM: text transcript
    LLM->>TOOL: get_account_balance(user_id=...)
    TOOL-->>LLM: { balance: "$1,240.50" }
    LLM->>TTS: "Your balance is $1,240.50"
    TTS->>UA: synthesized audio

In native S2S, tool calls are handled through a text-based side channel even in models that support them. The audio model signals a tool call, the system extracts it, calls the API, returns the result as text, and the model continues. This works — OpenAI's Realtime API supports it — but it reintroduces a text intermediate specifically for tool calls, which limits the "purity" of the native approach. For agents that are mostly tool-calling (scheduling bots, CRM-integrated support agents, appointment booking), the cascaded pipeline is architecturally cleaner.

Answer quality: whose brain is behind the voice

The quietest argument for cascaded is also the strongest one: the LLM in the middle can be any text model. If your agent needs frontier-grade reasoning — policy-heavy support decisions, multi-constraint scheduling, precise tool sequences — you point the pipeline at the strongest text model you can afford, and when a better one ships you swap it in that afternoon. The voice layer doesn't care.

Native S2S ties the conversation's intelligence to the speech model itself, and as of mid-2026 the speech-native variants measurably trail their text siblings on complex instruction-following, multi-step reasoning, and tool-call accuracy. That is not a permanent law, but it follows from how these models are built: post-training budget spent on sounding natural is budget not spent on being right, and the realtime constraint discourages the long deliberation that reasoning models use to earn their accuracy. If your agent's hardest problem is what to say rather than how to say it, cascaded wins before latency or cost ever enter the discussion.

Telephony is a different problem

PSTN telephone networks use 8 kHz audio with G.711 or G.729 codec compression. Web apps use 16 kHz Opus over WebRTC. The quality difference is significant, and native S2S models trained on high-quality audio take a noticeable hit on telephone-grade input.

The practical evidence: telephony deployments that switched from cascaded to native S2S and back have found that cascaded pipelines with STT models specifically trained on telephone audio (Deepgram is the clearest example, with explicit telephony model variants) outperform native S2S on phone call quality. The emotional prosody advantage of native S2S is also largely moot over phone audio — the codec compression strips most of the prosodic information before it ever reaches the model.

And the native S2S pricing does not discount for telephone audio quality. You still pay $0.30/min for gpt-realtime-1.5 whether the audio is pristine WebRTC or 8 kHz PSTN.

The voice pipeline article has a full breakdown of WebRTC vs SIP transport choices. For telephony deployments specifically, the architectural verdict is simple: use cascaded, use a telephony-optimized STT model, and save the native S2S experiments for your web app.

Where native S2S genuinely wins

This is not a blanket argument for cascaded. Native S2S has a real advantage category, and it is not latency — it is emotional and vocal continuity.

A cascaded pipeline destroys vocal context at every hand-off. The STT model converts the user's tired, anxious, or excited speech to neutral ASCII text. The LLM responds to that neutral text. The TTS model synthesizes a response with whatever expressiveness its defaults allow, with no connection to how the user actually sounded. The voice output is vocally decoupled from the voice input.

Native S2S maintains the acoustic thread. The model "heard" the user's hesitations, their raised pitch, their trailing-off cadence. That information shapes how it responds. Hume EVI 3 takes this furthest: it is the only production model that explicitly tracks emotional prosody and generates responses calibrated to the user's detected emotional state. This is not a gimmick for consumer apps — it is meaningful for healthcare consultations, mental health support tools, and any use case where the quality of the interaction depends on emotional attunement.

For a consumer voice assistant where "it feels like a real conversation" is the core product value, native S2S is worth the cost premium and the transcript constraint. For a scheduling bot handling appointment reminders at 50,000 calls per month, it is not.

flowchart TD
    START[Voice agent project] --> PHONE{Phone / PSTN?}
    PHONE -->|Yes| CASCADE[Cascaded pipeline\nwith telephony STT]
    PHONE -->|No| COMPLY{Compliance logging\nor tool-heavy?}
    COMPLY -->|Yes| CASCADE
    COMPLY -->|No| VOLUME{Volume > 50K min/month?}
    VOLUME -->|Yes| COST{Can absorb\n$0.30/min?}
    COST -->|No| CASCADE
    COST -->|Yes| EMOTION{Emotional prosody\nor empathy-critical?}
    VOLUME -->|No| EMOTION
    EMOTION -->|Yes| NATIVE[Native S2S\nHume EVI 3 or OpenAI Realtime]
    EMOTION -->|No| FEEL{Conversation feel\nis the core product?}
    FEEL -->|Yes| NATIVE
    FEEL -->|No| CASCADE

    style CASCADE fill:#0e7490,color:#fff
    style NATIVE fill:#00e5ff,color:#0a0a0f

Cost math at scale

The billing difference between architectures compounds quickly. Here is the calculation for a customer service voice agent at different volumes, using illustrative mid-2026 pricing:

Assumptions:
  Average call: 4 minutes
  LLM tokens: ~80 output/min; effective input ~8–12K tokens/min,
    because every turn resends the growing conversation history
    (fresh speech alone is only ~200 tokens/min)
  Cascaded: Deepgram Nova-3 + GPT-4o + Cartesia Sonic 4
  Native: OpenAI gpt-realtime-1.5

--- 10,000 min/month (small deployment) ---

Cascaded:
  STT  10,000 × $0.005 =    $50
  LLM  10,000 × $0.030 =   $300  (history-heavy input dominates;
                                  prefix caching can cut this 2–4×)
  TTS  10,000 × $0.028 =   $280
  Total:                    $630/month

Native S2S:
  10,000 × $0.30 =        $3,000/month

Gap: $2,370/month — probably not the deciding factor at this volume.

--- 100,000 min/month (mid-scale) ---

Cascaded: ~$6,300/month
Native S2S: ~$30,000/month
Gap: $23,700/month → $284,400/year

--- 500,000 min/month (large deployment) ---

Cascaded: ~$31,500/month
Native S2S: ~$150,000/month
Gap: $118,500/month → $1,422,000/year

Below 10,000 minutes per month, the absolute cost difference is manageable and should not be the deciding factor. Above 50,000 minutes per month, it is the primary architectural concern. Above 200,000 minutes per month, native S2S requires a specific product justification that cost cannot override.

Amazon Nova Sonic changes this calculation at the low end: at ~$0.02/min it is 15× cheaper than gpt-realtime-1.5, though it carries a ~1.14s TTFA vs the ~820ms of OpenAI's model. For use cases that can absorb the higher latency, Nova Sonic is the native S2S option that makes cost math work at larger volumes.

What breaks

The echo feedback spiral

Without acoustic echo cancellation (AEC), the agent's outgoing audio leaks into the user's microphone through their speakerphone. The STT model transcribes the agent's own voice, the LLM receives its own output as user input, and the agent starts responding to itself. This happens in both architectures, but in cascaded it is more immediately visible because the transcript shows the problem. In native S2S, the model may silently incorporate its own audio output into its next response without an obvious text artifact.

Twilio's server-side AEC works for good acoustics; it fails on cheap speakerphones and poor room conditions. If your deployment targets mobile users or call centers, AEC is a non-negotiable first-class concern, not an afterthought.

Turn detection misfires on silence

Both architectures depend on voice activity detection (VAD) to know when a user has finished speaking. Energy-based VAD fires on silence thresholds — it treats any pause as a turn boundary. Users who pause mid-thought, hesitate on a word, or use filler speech get cut off. In native S2S this produces an especially jarring result because the model's response is already emotionally calibrated to a half-finished input.

OpenAI's Semantic VAD mode — which waits for semantic completeness rather than mere silence — significantly reduces false-positive interruptions but adds roughly 100ms to the turn detection latency. The VAD and turn detection article covers the mechanics and tradeoffs in detail. The point here: semantic VAD is important for both architectures, but it is especially important for native S2S where a premature turn-end produces a contextually inappropriate emotional response.

Barge-in: cancelling a response already in flight

The user interrupts mid-answer — the single most common event in a real voice conversation — and both architectures have to unwind work that is already streaming. In a cascaded pipeline that means cancelling three things at once: the in-flight LLM generation, the TTS synthesis it was feeding, and the audio sitting in the playback buffer. Then comes the part teams get wrong: reconciling conversation state. If the agent had generated three sentences but the user heard one and a half, the history must be truncated to what was actually played — track playback position, cut the assistant turn at that point — or the model will keep referring to things the user never heard.

Native S2S handles interruption detection in-model, which is genuinely nicer: the model stops on its own instead of waiting for your cancellation plumbing. But the state-reconciliation problem does not go away. You still have to tell the server how much audio the user actually heard (OpenAI's Realtime API exposes a truncation call for exactly this), and getting the truncation point wrong is a known pain point — the model's memory of its own turn silently diverges from the user's. Frameworks help here; LiveKit Agents runs the cascaded cancellation loop for you. But test barge-in explicitly in either architecture, with a user who interrupts rudely and often. Production users do.

Debugging without a transcript

When a cascaded pipeline produces a bad response, you have a complete paper trail: the STT transcript, the LLM input, the LLM output, the latency at each stage. When a native S2S model produces a bad response, you have audio in and audio out. Reproducing the failure requires re-playing the audio; attributing it to input quality vs model behavior requires instrumentation that native S2S does not naturally provide.

This is a real operational cost for teams that take support escalations or run conversation quality reviews. The OpenAI Realtime API's async transcript side channel helps, but it is not available mid-turn and is not a substitute for the per-stage observability that cascaded provides.

Component lock-in

A cascaded pipeline lets you swap STT, LLM, and TTS independently. If Deepgram's latency spikes, you re-route to AssemblyAI. If GPT-4o costs too much for your token density, you switch to Gemini Flash. If ElevenLabs releases a better voice for your brand, you swap it in. Each component evolves independently.

Native S2S is a single vendor decision. If OpenAI degrades Realtime quality in a model update, you cannot swap the STT layer to compensate — you either accept the degradation or migrate the entire voice stack. For products where voice quality is a brand-level concern, this single-vendor risk is worth pricing in before you commit.

Mixing approaches

Nothing prevents a hybrid. A common production pattern: run native S2S for the main conversation flow (where emotional continuity matters), and spin up a cascaded STT side channel specifically for transcript-dependent operations. The side channel captures the user's speech in parallel and writes to your logging and CRM system. The native model handles the conversation feel.

The cost of this hybrid is running two inference pipelines simultaneously and the added complexity of keeping them in sync. It is worth considering for use cases that need both emotional prosody and a compliance log — but be honest that you are paying for both architectures, not getting the advantages of each for free.

sequenceDiagram
    participant U as User (audio)
    participant VAD as VAD
    participant S2S as Native S2S model
    participant STT as Parallel STT\n(compliance side channel)
    participant CRM as CRM / Audit log
    participant UA as User (audio out)

    U->>VAD: audio stream
    VAD->>S2S: speech segment
    VAD->>STT: same segment (parallel)
    S2S->>UA: audio response (emotional)
    STT->>CRM: text transcript (async)

The frameworks question

The choice of voice agent framework interacts with the architecture choice. Vapi and Retell AI are managed platforms that abstract both cascaded and native S2S behind their APIs — useful for low-volume deployments and fast iteration. LiveKit Agents 1.5.x and Pipecat v1.0 give you full control over the pipeline and support both architectures, with better economics above 50,000 minutes per month. The voice agent frameworks article has the decision matrix and cost thresholds.

One practical note: LiveKit's native S2S integration routes audio directly to the model using WebRTC transport and handles the barge-in cancellation loop for you. Pipecat's compositional architecture lets you wire native S2S as one "stage" in a pipeline that includes a separate STT side channel — which is the hybrid pattern described above. Neither framework forces your hand on the cascaded vs native decision; they support both.

The decision in practice

If you are picking between these architectures today, answer four questions in order:

1. Is PSTN telephony in scope? If yes, cascaded — unconditionally. Telephone-grade audio degrades native S2S quality while retaining its pricing. Use Deepgram Nova-3 with their telephony model variant.

2. Do you need a mid-turn transcript? Compliance logging, CRM writes, tool-call routing based on live transcript content, supervisor monitoring — any of these requires cascaded, or a hybrid with a parallel STT stream.

3. What is your monthly minute volume? Below 10,000 minutes, cost is not the deciding factor. Above 50,000 minutes, native S2S at $0.30/min requires a product justification that overrides the cost differential. Amazon Nova Sonic at $0.02/min is the exception that changes this threshold.

4. Is emotional prosody or conversation feel the core product value? Healthcare, mental health, high-empathy consumer assistants — go native. Hume EVI 3 specifically if emotional attunement is the product. For a scheduling bot or FAQ agent, voice quality is important but "sounds like a real person" is not the product differentiator; cascaded with ElevenLabs Flash v2.5 or Cartesia Sonic 4 sounds excellent and costs a fraction of the price. Weigh this against answer quality: if the agent's value lives in getting hard answers right, cascaded lets you put a frontier text model behind the voice, while native S2S caps reasoning at what the speech model can do.

The default recommendation for most enterprise and telephony deployments is cascaded with streaming enabled end-to-end, Deepgram for STT, a mid-tier frontier model for the LLM (with prompt prefix caching enabled), and Cartesia Sonic 4 for TTS. That stack, properly optimized, hits 500–700ms TTFA — matching or beating most native S2S models at a fifth of the cost or less.

Native S2S is the right answer for a specific product profile: consumer-facing, conversation-quality-is-the-product, emotional attunement matters, volume is manageable, and compliance teams are not in the room. As the models improve and costs come down, that profile will expand. But the engineering decision today requires being honest about where the production numbers actually land, not where the marketing says they should.

For the latency budget breakdown and optimization techniques that apply to both architectures, see latency budget engineering.