VAD and Turn Detection: The Hardest UX Problem in Voice AI

A customer calls your voice agent. Mid-sentence they take a breath, just two beats of silence while they collect the next thought. The agent fires. It starts talking over them. They say "wait, I wasn't done." The agent keeps going because it already generated the response and the TTS buffer is half-drained. The customer hangs up. This sequence happens thousands of times a day in production voice deployments, and it comes down to a single 200ms silence threshold in the VAD configuration.

The voice pipeline gets most of the engineering attention — faster STT, lower LLM TTFT, sub-50ms TTS. Turn detection is the component that receives the least attention in demos and causes the most damage in production. Getting it right requires understanding three distinct problems that are often conflated: detecting speech onset, detecting speech end, and handling the moment when both sides speak at once.

What VAD actually does

Voice activity detection splits an incoming audio stream into speech frames and non-speech frames. That sounds like a solved problem, and the core mechanics are indeed well-understood. The difficulty is in the decision boundary for end-of-turn, which determines when the downstream pipeline should act.

A VAD component runs on chunked audio, typically 10–30ms frames. For each frame it outputs either "speech" or "silence." The speech-end decision is not a single frame — it's a holdout: how many consecutive silence frames before you declare the turn over? That holdout value, measured in milliseconds, is where most of the UX lives.

sequenceDiagram
    participant U as User audio
    participant V as VAD
    participant P as Pipeline

    U->>V: speech frames (300ms)
    U->>V: silence frame #1 (30ms)
    U->>V: silence frame #2 (30ms)
    Note over V: energy-based VAD at 200ms threshold — still holding
    U->>V: silence frame #3 (30ms)
    U->>V: silence frame #4 (30ms)
    Note over V: energy-based VAD at 200ms threshold — still holding
    U->>V: silence frame #5 (30ms) → 150ms total silence
    U->>V: silence frame #6 (30ms)
    U->>V: silence frame #7 (30ms) → 210ms total silence
    Note over V: 200ms silence threshold crossed — fires end-of-turn
    V->>P: speech_stopped event
    U->>V: speech frame — user resumed mid-thought
    Note over P: already dispatching LLM call — false positive

The holdout is the tradeoff dial. A short holdout (100–200ms) means fast response time for crisp one-word answers, but it fires on the natural pauses in any complex sentence. A long holdout (500–800ms) tolerates pauses but makes the agent feel unresponsive on quick questions. Neither value is universally correct, and any fixed value will be wrong for some users.

Energy-based VAD

"Energy-based" is the colloquial label, and the truly trivial version does exist: compare the RMS energy of each audio frame to a decibel threshold, apply the holdout, and emit the event. Almost nobody ships that. What production systems deploy under this label are small acoustic frame classifiers — the end-of-turn decision is still "N ms of non-speech frames," but the per-frame call is a model, not an energy gate, which is why these tools survive background noise that would fool a raw dB threshold. Silero VAD is the dominant open-source option — a 1MB ONNX model that runs on CPU and emits per-chunk speech probabilities at 30ms resolution with ~1ms inference time per chunk. WebRTC's built-in VAD (included in most browser and mobile SDKs) is a GMM-based subband classifier working at 10ms resolution with comparable speed.

from silero_vad import load_silero_vad, read_audio, get_speech_timestamps

model = load_silero_vad()

# Process a 16kHz mono audio buffer
wav = read_audio("user_turn.wav", sampling_rate=16000)
speech_timestamps = get_speech_timestamps(
    wav,
    model,
    threshold=0.5,          # speech probability threshold
    sampling_rate=16000,
    min_silence_duration_ms=200,   # end-of-turn holdout
    min_speech_duration_ms=100,    # ignore very short noise bursts
)
# Returns list of {start: int, end: int} in samples

Silence-duration endpointing works well when speech is clean, the user is a native speaker who pauses at sentence boundaries, and there is no background noise. In a production call center, none of those conditions hold.

Semantic VAD

OpenAI shipped Semantic VAD in the Realtime API in 2025. Instead of firing on silence duration, the model waits until the transcript of the current utterance appears linguistically complete before signaling end-of-turn. A sentence ending in a falling intonation contour with a verb phrase intact is complete. "So I was thinking that maybe we could—" with a 300ms pause is not.

The implementation is a turn-prediction model that runs on the partial transcript. It adds approximately 100ms to end-of-turn latency — the model has to process the partial text and emit a confidence score before the pipeline acts. That 100ms cost is almost always worth it: the false-positive rate on mid-sentence pauses drops dramatically, and user satisfaction scores follow.

You configure turn detection in the OpenAI Realtime API via the turn_detection field on session creation. As of mid-2026 there are two modes: server_vad (silence-based, with configurable silence_duration_ms and threshold) and semantic_vad (the turn-prediction model described above, tuned via an eagerness setting). These field names have shifted between Realtime releases, so pin your API version:

import openai

client = openai.OpenAI()

# Create a Realtime session with semantic turn detection (as of mid-2026)
session = client.beta.realtime.sessions.create(
    model="gpt-realtime-1.5",
    turn_detection={
        "type": "semantic_vad",   # or "server_vad" for silence-based endpointing
        "eagerness": "medium",    # low = tolerate longer pauses; high = respond faster
    },
    input_audio_format="pcm16",
    output_audio_format="pcm16",
)

In server_vad mode, silence_duration_ms is the equivalent of the silence holdout: longer values tolerate more pause (better for complex instructions), shorter values fire faster (better for rapid back-and-forth). Pin to a specific API version in production, as these parameters have changed between Realtime API releases.

LiveKit Agents 1.5.x (April 2026) ships its own adaptive interruption logic on top of whatever STT/VAD you configure — it adjusts sensitivity based on how the conversation has gone, becoming less trigger-happy if it has already produced false positives earlier in the call.

{ "type": "voice-latency", "title": "VAD mode impact on perceived latency" }

The barge-in problem

Barge-in is when the user starts speaking while the agent is mid-response. VAD detects the user's speech onset, fires a speech_started event, and the pipeline must react.

The naive implementation ignores it. The agent keeps playing audio while the user is talking, the STT quietly transcribes the new speech, and after the agent finishes it processes the new turn. This produces the walkie-talkie effect — both parties wait for the other to fully finish — which feels deeply unnatural in voice conversation.

A correct barge-in implementation requires three coordinated steps in tight sequence:

sequenceDiagram
    participant U as User
    participant V as VAD
    participant LL as LLM (in-flight)
    participant TT as TTS buffer
    participant SPK as Speaker

    SPK->>U: agent playing audio...
    U->>V: starts speaking
    V->>LL: speech_started → cancel generation
    V->>TT: flush buffered audio immediately
    V->>SPK: stop playback
    Note over LL,TT: must complete within ~50–150ms or user hears overlap
    U->>V: continues speaking
    V->>STT: start new transcription
    STT->>LL: new turn begins

Step 1 is canceling the LLM generation. If you're using a streaming API, this means calling the cancellation endpoint or closing the stream. Any tokens the model generates after the cancel are wasted, but more importantly, a goroutine or thread is pinned to that in-flight request until it completes. At scale — thousands of concurrent calls — this becomes a resource leak. Always cancel promptly.

Step 2 is discarding the TTS buffer. TTS synthesis typically runs ahead of playback by a few hundred milliseconds. When barge-in fires, any buffered audio that hasn't played yet should be discarded. If it isn't, the agent's old response continues playing for the duration of the buffer drain — often 200–500ms of unwanted overlap on top of the user's new speech.

Step 3 is stopping the audio output. This is usually a call to the WebRTC or audio device layer to halt the current stream.

The latency budget for these three steps is tight: if the combined cancellation takes more than 150ms, users hear the old response playing over their new question. Most well-optimized implementations target 50–100ms for the full cancellation cycle.

Here is a simplified barge-in handler in the style of a LiveKit or Pipecat pipeline:

import asyncio
from dataclasses import dataclass
from typing import Optional

@dataclass
class AgentTurn:
    llm_task: asyncio.Task
    tts_buffer: asyncio.Queue
    audio_output: "AudioOutputStream"

class BargeInHandler:
    def __init__(self):
        self.current_turn: Optional[AgentTurn] = None

    async def on_speech_started(self, event: dict) -> None:
        """VAD fired speech_started — cancel current agent turn if active."""
        if self.current_turn is None:
            return  # agent wasn't speaking, nothing to cancel

        turn = self.current_turn
        self.current_turn = None

        # Step 1: cancel LLM generation — stop wasting tokens
        if not turn.llm_task.done():
            turn.llm_task.cancel()
            try:
                await asyncio.wait_for(turn.llm_task, timeout=0.1)
            except (asyncio.CancelledError, asyncio.TimeoutError):
                pass  # expected

        # Step 2: drain the TTS buffer — don't play stale audio
        while not turn.tts_buffer.empty():
            try:
                turn.tts_buffer.get_nowait()
            except asyncio.QueueEmpty:
                break

        # Step 3: stop audio output immediately
        await turn.audio_output.stop()

    async def on_agent_response_started(self, turn: AgentTurn) -> None:
        self.current_turn = turn

The OpenAI Realtime API handles barge-in internally when using server-managed sessions — the input_audio_buffer.speech_started event signals that barge-in has been detected, and the API automatically cancels the current response and emits response.cancelled. You still need to stop your local audio playback.

Echo cancellation: the prerequisite nobody talks about

Before VAD can work correctly, you need acoustic echo cancellation. Without AEC, the microphone picks up the agent's outgoing audio through the speaker, the STT transcribes it, and the LLM receives its own words as a new user turn. The agent responds to itself. The loop tightens. Nobody's done what you wanted.

Browsers provide AEC via the WebRTC stack automatically when you use getUserMedia with echoCancellation: true. Mobile native apps have OS-level AEC. The problem cases are:

• Cheap speakerphones: the physical isolation between speaker and mic is poor, and the echo leakback amplitude is high enough to saturate the AEC filter.
• Landline / PSTN calls: the PSTN network path introduces variable delay that misaligns the echo reference signal the AEC algorithm relies on.
• Open-plan offices: multi-path room reflections confuse single-microphone AEC.

Twilio's server-side AEC is the common managed option for telephony, but it fails on cheap speakerphones and poor room acoustics. For high-stakes telephony deployments (healthcare, financial services), dedicated telephony AEC hardware or a service like Krisp's noise cancellation layer is worth evaluating.

The rule is simple: confirm AEC is working before you start tuning VAD thresholds. A 30-second recording of the agent talking to itself is diagnostic.

What breaks

Mid-pause false cuts

The single most-cited UX complaint. A user says: "I'd like to dispute a charge on my account — let me look up the amount." The pause after "account" is 400ms. Energy VAD at 300ms fires. The agent asks "what charge?" before the user has finished their context. The user restarts from scratch, the agent re-asks the question, and the experience degrades into mutual confusion.

Mitigation: semantic VAD, or a longer energy threshold (500ms+). Longer threshold means slower response on clear short questions, so you're trading one bad experience for a slightly worse version of another. Semantic VAD is the better answer when your latency budget allows the extra 100ms.

Barge-in storms

In a noisy environment — a call center floor, a car on a highway — intermittent background noise triggers repeated speech-started events even when the user isn't talking. Each event fires the barge-in cancellation cycle. The agent never gets more than a few seconds into a response before being canceled. The call goes nowhere.

Mitigation: noise suppression (RNNoise, Krisp, or WebRTC's built-in NS) upstream of the VAD, plus a minimum speech duration threshold (100ms minimum keeps random pops from triggering full cancellations).

Double-talk and overlapping speech

When user and agent genuinely speak simultaneously — not barge-in, but actual simultaneous overlapping speech — transcription quality falls off a cliff. Interspeech research measured 16.8% WER on clean single-speaker audio versus 74.6% WER on overlapping speech. That's not a marginal degradation; it means roughly three-quarters of overlapping words are wrong.

Production pipelines handle this by serializing: when barge-in fires, stop the agent and process the user's next turn after. This produces correct-but-unnatural conversation that feels like a protocol rather than a dialogue. True full-duplex voice is an open research problem.

TTS buffer drain timing

A less obvious failure mode: the TTS synthesis pipeline runs ahead of playback. When barge-in fires, if you drain the buffer but don't abort in-progress TTS synthesis, the synthesis worker continues filling the buffer and the old content bleeds back in. Make sure your cancellation propagates all the way up the TTS call stack, not just to the buffer reader.

Silent users and infinite wait

Energy VAD waits for silence. If the user is thinking and stays silent for several seconds, the VAD either fires prematurely (short threshold) or never fires at all until you add a maximum silence timeout. Set a maximum turn silence ceiling — typically 10–15 seconds — after which the agent prompts ("Are you still there?"). Without this, a user who puts the call on hold can leave the pipeline hanging indefinitely.

Worked latency budget: energy vs semantic VAD

Session: browser client, Deepgram Nova-3 STT, gpt-4o-mini LLM, Cartesia Sonic 4 TTS

Energy VAD (200ms holdout):
  STT latency from speech end: 150ms (Deepgram streaming, flush on end-of-turn)
  LLM TTFT: 220ms
  TTS TTFB: 40ms (Cartesia Sonic 4)
  Network RTT: 80ms (US East, co-located)
  Total TTFA from speech end: ~490ms ✓ (sub-500ms target)
  False positive rate on real conversations: ~12% (measured on 200 calls)

Semantic VAD (100ms model overhead):
  VAD overhead: +100ms
  STT latency from speech end: 150ms
  LLM TTFT: 220ms
  TTS TTFB: 40ms
  Network RTT: 80ms
  Total TTFA from speech end: ~590ms ✗ (just over 500ms target)
  False positive rate: ~1.5%

Decision: the 100ms overage is acceptable given the 8× reduction in false positives.
Users perceive 590ms as slightly slow but tolerable; they perceive being cut off as hostile.

The math shows why this is a genuine tension. Semantic VAD produces meaningfully better conversations but pushes past the 500ms perception threshold that separates "natural" from "slightly slow." For a voice agent where turn quality matters more than raw latency (support escalation, healthcare triage), semantic is the right call. For a high-volume outbound sales dialer optimizing for rapid yes/no responses, energy VAD at a short holdout might win.

VAD in native speech-to-speech pipelines

When you use a native S2S model — OpenAI's gpt-realtime-1.5, Gemini 3.1 Flash Live, or Amazon Nova Sonic — the turn detection is internal to the model. You don't wire up a separate VAD component; instead you configure turn-detection behavior on the session and respond to the events the API emits.

This simplifies the architecture but reduces control. You can't swap in Silero VAD tuned to your noise floor, you can't run your own AEC preprocessing (you're sending raw audio to the API), and the debug story for "why did the agent cut me off?" is opaque. Native S2S models also cost significantly more — OpenAI Realtime at ~$0.30/min versus a cascaded pipeline with Deepgram + gpt-4o-mini + Cartesia at roughly $0.04–0.08/min — so every false barge-in is more expensive to produce.

For the full comparison between cascaded and native architectures, see the sibling article on cascaded vs native speech-to-speech.

Tuning in practice

The right workflow is empirical, not theoretical:

1. Record 100+ real user conversations from a pilot or beta. Include your actual noise environment (office, car, call center).
2. Measure false-positive barge-in rate: count events where the VAD fired during a mid-sentence pause and the user hadn't finished. Target under 5%.
3. Measure missed barge-in rate: count events where the user tried to interrupt and the pipeline didn't respond within 300ms. Target under 10%.
4. Tune threshold on this corpus, not the vendor's clean-studio defaults.
5. Re-measure after any infrastructure change — codec changes, network path changes, and hardware changes all shift the noise floor.

Deepgram and AssemblyAI both offer streaming VAD tuning parameters (utterance_end_ms in Deepgram, end_utterance_silence_threshold in AssemblyAI). Treat these as starting points. The vendor's defaults are tuned for their benchmark test suite, which does not resemble your production audio profile. Independent benchmarks from Coval show 15–30% WER degradation in real call-center noise compared to vendor benchmarks — and VAD tuning degrades proportionally.

flowchart TD
    START[Deploy with vendor defaults] --> RECORD[Record 100+ real calls]
    RECORD --> MEASURE[Measure false-positive rate]
    MEASURE --> HIGH{">5% false positives?"}
    HIGH -->|yes| LONGER[Increase silence threshold\nor switch to semantic VAD]
    HIGH -->|no| SLOW{">10% missed barge-ins?"}
    SLOW -->|yes| SHORTER[Decrease threshold\nor increase eagerness]
    SLOW -->|no| DONE[Ship — re-measure after\nany infra change]
    LONGER --> MEASURE
    SHORTER --> MEASURE
    style HIGH fill:#00e5ff,color:#0a0a0f
    style SLOW fill:#ffaa00,color:#0a0a0f
    style DONE fill:#15803d,color:#fff

When VAD is not your problem

If users complain about the agent being slow to respond, that's almost certainly latency budget, not VAD. See latency budget engineering for the full breakdown.

If users complain about the agent ignoring questions or answering the wrong thing, that's more likely STT accuracy or LLM context management, not turn detection. VAD firing at the right time doesn't help if the transcript is wrong or the context window is full of stale history.

If users complain that the agent doesn't understand their accent, that's STT model selection. Deepgram Nova-3 and AssemblyAI Universal-2 handle accented speech better than Whisper; the voice pipeline overview covers the full provider comparison.

VAD is specifically the problem when users describe being "cut off," when the agent "doesn't let me finish," or when the agent "talks over itself." Those symptom descriptions are the diagnostic signal.

The decision in practice

Use energy-based VAD when:

• Your latency budget is under 500ms and every millisecond counts (outbound dialer, rapid confirmation flows)
• Your audio is clean and your users are concise (kiosk, IVR replacement)
• You're using native S2S anyway and the model handles turn detection internally
• Your volume is high and you're paying per minute on a managed platform — false positives cost money

Use semantic VAD when:

• Your use case involves complex user instructions or hesitant speech (support, healthcare, financial advice)
• False-positive barge-ins are your top complaint and you have room to absorb ~100ms additional latency
• You're on OpenAI Realtime and can set turn_detection.type: "semantic_vad" on the session
• You have heard "it keeps cutting me off" from more than 5% of users in beta

Side by side, using the numbers from the worked budget above (illustrative, mid-2026 pricing):

	Silence-based VAD (Silero, WebRTC, server_vad)	Semantic VAD	Native S2S internal (Realtime, Gemini Live)
End-of-turn latency overhead	~0ms beyond the holdout itself	~100ms turn-prediction model	Opaque — vendor-tuned, not separately measurable
Mid-pause false-positive rate	~12% untuned on real calls	~1.5%	Generally low, but you can't instrument it
Tunability	Full: threshold, holdout, min-speech duration	Coarse: an eagerness-style dial	Session config only
AEC / preprocessing control	Full — your pipeline, your noise suppression	Full upstream of the turn model	None — raw audio goes to the API
Cost of each false barge-in	One wasted STT flush + LLM call (~fractions of a cent)	Rare; same waste when it fires	Highest — ~$0.30/min platform pricing vs ~$0.04–0.08/min cascaded

The honest assessment: semantic VAD should be the default for almost all conversational agents. The 100ms additional latency is a fair price for not interrupting users mid-thought, and it's recoverable through other optimizations — streaming LLM tokens into TTS, regional inference routing, prompt prefix caching. Being cut off mid-sentence is not recoverable through anything except not doing it.

The voice pipeline frameworks (Vapi, LiveKit, Pipecat) all expose VAD configuration at different levels of granularity. Vapi abstracts most of it. LiveKit Agents and Pipecat give you direct access to the parameters. If you're at the stage where VAD tuning matters, you're probably past the point where Vapi's defaults are sufficient anyway.

Turn detection is not a solved problem, and the field has not produced a universally correct answer. The 100ms semantic overhead will shrink as models get faster. Double-talk handling remains an open research area. What is solved: the causes of the most common failures, the parameters that control them, and the measurement approach that tells you whether your configuration is working. Start there.