# VAD and Turn Detection: The Hardest UX Problem in Voice AI > How voice activity detection and turn-taking work in real-time AI agents — energy vs semantic VAD, barge-in handling, and why getting this wrong is the top user complaint. Canonical: https://ironclad.academy/ai-engineering/articles/vad-turn-detection-voice-ai | Difficulty: medium | Topics: Voice, Latency, Streaming, Production Covers: OpenAI, LiveKit, Deepgram, ElevenLabs ## Key takeaways - Energy-based VAD cuts users off mid-sentence; semantic VAD trades ~100ms of extra latency for dramatically fewer false-positive barge-ins. - A barge-in event requires canceling the LLM call and discarding buffered TTS audio — skipping either step wastes compute and produces jarring audio overlap. - Overlapping speech (double-talk) raises word error rate from ~17% to ~75% — production pipelines serialize turn-taking rather than truly handling it in parallel. - Tune VAD on your actual audio profile (call-center noise, speakerphone distortion) not vendor clean-studio benchmarks, which test conditions you will never see in production. - Echo cancellation prevents the agent from transcribing its own voice but does not give you full-duplex conversation — those are separate problems with separate solutions. - The barge-in cancellation budget is typically 50–150ms; anything slower and the user hears the old response overlapping their new question. > **In one line:** VAD and turn detection decide when the user has stopped talking — get it wrong and your voice agent either cuts people off or waits in awkward silence, and no amount of faster LLM inference will fix it. **The problem.** Every voice AI pipeline needs to know when to listen and when to respond. That boundary — the end of a user's turn — sounds trivial and turns out to be surprisingly hard. Energy-based detection fires on silence, including mid-sentence pauses and filler words. The agent interrupts the user. Users hate it and stop using the product. The fix, semantic VAD, adds latency. Barge-in handling requires three coordinated cancellation steps that, if any one fails, produce garbled audio or wasted compute. And double-talk (user and agent speaking simultaneously) degrades transcription accuracy so severely that most production systems just avoid it entirely. **Key points.** - **Energy-based VAD** fires after N ms of silence below a dB threshold — fast, zero model overhead, but triggers on mid-sentence pauses and hesitations. - **Semantic VAD** runs a model that predicts linguistic completeness before ending the turn — ~100ms slower, dramatically fewer false cuts. - **Barge-in** requires three coordinated steps: cancel LLM, discard TTS buffer, restart STT. Budget 50–150ms total or the user hears overlap. - **Double-talk** (simultaneous speech) raises WER from ~17% to ~75%; production systems serialize turns rather than handle it in parallel. - **Acoustic echo cancellation (AEC)** is a prerequisite, not VAD itself — without it, the agent transcribes its own voice and enters a feedback spiral. ``` Back-of-envelope: VAD timing budget ───────────────────────────────────────────────────────── User pauses 300ms mid-thought. Energy VAD at 200ms threshold → fires 200ms into that pause. Pipeline dispatches STT flush + LLM call. User resumes 100ms later. Barge-in fires → cancel LLM, discard TTS synthesis in flight. Cost: one wasted LLM call, a discarded turn, and a stutter while the pipeline restarts on the user's continued sentence. Semantic VAD at same point: Model predicts utterance incomplete → holds for 100ms more. User resumes → no false positive. Cost: 100ms extra latency on real end-of-turn events. Benefit: 0 false barge-ins on this class of pause. Rule of thumb: tune energy VAD threshold to <5% false-positive rate on a sample of real user recordings before shipping. ``` ```mermaid flowchart LR MIC[Microphone audio] --> AEC[Echo Cancellation] AEC --> VAD{VAD decision} VAD -->|speech detected| STT[STT — stream partial transcripts] VAD -->|end of turn| TURN[Turn signal to pipeline] TURN --> LLM[LLM inference] LLM -->|streaming tokens| TTS[TTS synthesis] TTS --> SPK[Speaker output] SPK -.->|leakback without AEC| MIC style VAD fill:#00e5ff,color:#0a0a0f style AEC fill:#15803d,color:#fff ``` **Key design decisions.** | Decision | Choice | Why | |---|---|---| | VAD mode | Semantic (if latency budget allows) | Eliminates mid-pause false cuts — the #1 user complaint | | Silence threshold | 200–400ms at –40 to –50 dBFS | Tuned on your audio profile, not vendor defaults | | Barge-in cancellation | Cancel LLM + drain TTS buffer + stop audio | Partial cancellation wastes tokens or plays garbled audio | | Echo cancellation | Server-side AEC or WebRTC built-in | Prerequisite for any other VAD decision to work | | Double-talk handling | Serialize (queue new turn) | True parallel handling not viable at production WER | **If you have 60 seconds, say this.** "VAD is the component that draws the line between user speaking and agent responding. Energy-based VAD fires on silence — cheap but cuts users off mid-sentence. Semantic VAD waits for the utterance to be linguistically complete, adding about 100ms but eliminating most false positives. When the user speaks during agent playback, a barge-in event must cancel the current LLM call, discard buffered TTS audio, and start fresh — any step missed and you get wasted compute or garbled audio. Echo cancellation has to come first or the agent transcribes its own output. And overlapping speech is effectively unsolved — WER goes from 17% to 75%, so production systems just serialize turns like a walkie-talkie and the user experience reflects that." A customer calls your voice agent. Mid-sentence they take a breath, just two beats of silence while they collect the next thought. The agent fires. It starts talking over them. They say "wait, I wasn't done." The agent keeps going because it already generated the response and the TTS buffer is half-drained. The customer hangs up. This sequence happens thousands of times a day in production voice deployments, and it comes down to a single 200ms silence threshold in the VAD configuration. The voice pipeline gets most of the engineering attention — faster STT, lower LLM TTFT, sub-50ms TTS. Turn detection is the component that receives the least attention in demos and causes the most damage in production. Getting it right requires understanding three distinct problems that are often conflated: detecting speech onset, detecting speech end, and handling the moment when both sides speak at once. ## What VAD actually does **Voice activity detection** splits an incoming audio stream into speech frames and non-speech frames. That sounds like a solved problem, and the core mechanics are indeed well-understood. The difficulty is in the decision boundary for end-of-turn, which determines when the downstream pipeline should act. A VAD component runs on chunked audio, typically 10–30ms frames. For each frame it outputs either "speech" or "silence." The speech-end decision is not a single frame — it's a holdout: how many consecutive silence frames before you declare the turn over? That holdout value, measured in milliseconds, is where most of the UX lives. ```mermaid sequenceDiagram participant U as User audio participant V as VAD participant P as Pipeline U->>V: speech frames (300ms) U->>V: silence frame #1 (30ms) U->>V: silence frame #2 (30ms) Note over V: energy-based VAD at 200ms threshold — still holding U->>V: silence frame #3 (30ms) U->>V: silence frame #4 (30ms) Note over V: energy-based VAD at 200ms threshold — still holding U->>V: silence frame #5 (30ms) → 150ms total silence U->>V: silence frame #6 (30ms) U->>V: silence frame #7 (30ms) → 210ms total silence Note over V: 200ms silence threshold crossed — fires end-of-turn V->>P: speech_stopped event U->>V: speech frame — user resumed mid-thought Note over P: already dispatching LLM call — false positive ``` The holdout is the tradeoff dial. A short holdout (100–200ms) means fast response time for crisp one-word answers, but it fires on the natural pauses in any complex sentence. A long holdout (500–800ms) tolerates pauses but makes the agent feel unresponsive on quick questions. Neither value is universally correct, and any fixed value will be wrong for some users. ### Energy-based VAD "Energy-based" is the colloquial label, and the truly trivial version does exist: compare the RMS energy of each audio frame to a decibel threshold, apply the holdout, and emit the event. Almost nobody ships that. What production systems deploy under this label are small acoustic frame classifiers — the end-of-turn decision is still "N ms of non-speech frames," but the per-frame call is a model, not an energy gate, which is why these tools survive background noise that would fool a raw dB threshold. **Silero VAD** is the dominant open-source option — a 1MB ONNX model that runs on CPU and emits per-chunk speech probabilities at 30ms resolution with ~1ms inference time per chunk. WebRTC's built-in VAD (included in most browser and mobile SDKs) is a GMM-based subband classifier working at 10ms resolution with comparable speed. ```python from silero_vad import load_silero_vad, read_audio, get_speech_timestamps model = load_silero_vad() # Process a 16kHz mono audio buffer wav = read_audio("user_turn.wav", sampling_rate=16000) speech_timestamps = get_speech_timestamps( wav, model, threshold=0.5, # speech probability threshold sampling_rate=16000, min_silence_duration_ms=200, # end-of-turn holdout min_speech_duration_ms=100, # ignore very short noise bursts ) # Returns list of {start: int, end: int} in samples ``` Silence-duration endpointing works well when speech is clean, the user is a native speaker who pauses at sentence boundaries, and there is no background noise. In a production call center, none of those conditions hold. ### Semantic VAD OpenAI shipped **Semantic VAD** in the Realtime API in 2025. Instead of firing on silence duration, the model waits until the transcript of the current utterance appears linguistically complete before signaling end-of-turn. A sentence ending in a falling intonation contour with a verb phrase intact is complete. "So I was thinking that maybe we could—" with a 300ms pause is not. The implementation is a turn-prediction model that runs on the partial transcript. It adds approximately 100ms to end-of-turn latency — the model has to process the partial text and emit a confidence score before the pipeline acts. That 100ms cost is almost always worth it: the false-positive rate on mid-sentence pauses drops dramatically, and user satisfaction scores follow. You configure turn detection in the OpenAI Realtime API via the `turn_detection` field on session creation. As of mid-2026 there are two modes: `server_vad` (silence-based, with configurable `silence_duration_ms` and `threshold`) and `semantic_vad` (the turn-prediction model described above, tuned via an `eagerness` setting). These field names have shifted between Realtime releases, so pin your API version: ```python import openai client = openai.OpenAI() # Create a Realtime session with semantic turn detection (as of mid-2026) session = client.beta.realtime.sessions.create( model="gpt-realtime-1.5", turn_detection={ "type": "semantic_vad", # or "server_vad" for silence-based endpointing "eagerness": "medium", # low = tolerate longer pauses; high = respond faster }, input_audio_format="pcm16", output_audio_format="pcm16", ) ``` In `server_vad` mode, `silence_duration_ms` is the equivalent of the silence holdout: longer values tolerate more pause (better for complex instructions), shorter values fire faster (better for rapid back-and-forth). Pin to a specific API version in production, as these parameters have changed between Realtime API releases. LiveKit Agents 1.5.x (April 2026) ships its own adaptive interruption logic on top of whatever STT/VAD you configure — it adjusts sensitivity based on how the conversation has gone, becoming less trigger-happy if it has already produced false positives earlier in the call. [Interactive visualizer on the original page: voice-latency — VAD mode impact on perceived latency] ## The barge-in problem Barge-in is when the user starts speaking while the agent is mid-response. VAD detects the user's speech onset, fires a `speech_started` event, and the pipeline must react. The naive implementation ignores it. The agent keeps playing audio while the user is talking, the STT quietly transcribes the new speech, and after the agent finishes it processes the new turn. This produces the walkie-talkie effect — both parties wait for the other to fully finish — which feels deeply unnatural in voice conversation. A correct barge-in implementation requires three coordinated steps in tight sequence: ```mermaid sequenceDiagram participant U as User participant V as VAD participant LL as LLM (in-flight) participant TT as TTS buffer participant SPK as Speaker SPK->>U: agent playing audio... U->>V: starts speaking V->>LL: speech_started → cancel generation V->>TT: flush buffered audio immediately V->>SPK: stop playback Note over LL,TT: must complete within ~50–150ms or user hears overlap U->>V: continues speaking V->>STT: start new transcription STT->>LL: new turn begins ``` Step 1 is canceling the LLM generation. If you're using a streaming API, this means calling the cancellation endpoint or closing the stream. Any tokens the model generates after the cancel are wasted, but more importantly, a goroutine or thread is pinned to that in-flight request until it completes. At scale — thousands of concurrent calls — this becomes a resource leak. Always cancel promptly. Step 2 is discarding the TTS buffer. TTS synthesis typically runs ahead of playback by a few hundred milliseconds. When barge-in fires, any buffered audio that hasn't played yet should be discarded. If it isn't, the agent's old response continues playing for the duration of the buffer drain — often 200–500ms of unwanted overlap on top of the user's new speech. Step 3 is stopping the audio output. This is usually a call to the WebRTC or audio device layer to halt the current stream. The latency budget for these three steps is tight: if the combined cancellation takes more than 150ms, users hear the old response playing over their new question. Most well-optimized implementations target 50–100ms for the full cancellation cycle. Here is a simplified barge-in handler in the style of a LiveKit or Pipecat pipeline: ```python import asyncio from dataclasses import dataclass from typing import Optional @dataclass class AgentTurn: llm_task: asyncio.Task tts_buffer: asyncio.Queue audio_output: "AudioOutputStream" class BargeInHandler: def __init__(self): self.current_turn: Optional[AgentTurn] = None async def on_speech_started(self, event: dict) -> None: """VAD fired speech_started — cancel current agent turn if active.""" if self.current_turn is None: return # agent wasn't speaking, nothing to cancel turn = self.current_turn self.current_turn = None # Step 1: cancel LLM generation — stop wasting tokens if not turn.llm_task.done(): turn.llm_task.cancel() try: await asyncio.wait_for(turn.llm_task, timeout=0.1) except (asyncio.CancelledError, asyncio.TimeoutError): pass # expected # Step 2: drain the TTS buffer — don't play stale audio while not turn.tts_buffer.empty(): try: turn.tts_buffer.get_nowait() except asyncio.QueueEmpty: break # Step 3: stop audio output immediately await turn.audio_output.stop() async def on_agent_response_started(self, turn: AgentTurn) -> None: self.current_turn = turn ``` The OpenAI Realtime API handles barge-in internally when using server-managed sessions — the `input_audio_buffer.speech_started` event signals that barge-in has been detected, and the API automatically cancels the current response and emits `response.cancelled`. You still need to stop your local audio playback. ## Echo cancellation: the prerequisite nobody talks about Before VAD can work correctly, you need acoustic echo cancellation. Without AEC, the microphone picks up the agent's outgoing audio through the speaker, the STT transcribes it, and the LLM receives its own words as a new user turn. The agent responds to itself. The loop tightens. Nobody's done what you wanted. Browsers provide AEC via the WebRTC stack automatically when you use `getUserMedia` with `echoCancellation: true`. Mobile native apps have OS-level AEC. The problem cases are: - **Cheap speakerphones**: the physical isolation between speaker and mic is poor, and the echo leakback amplitude is high enough to saturate the AEC filter. - **Landline / PSTN calls**: the PSTN network path introduces variable delay that misaligns the echo reference signal the AEC algorithm relies on. - **Open-plan offices**: multi-path room reflections confuse single-microphone AEC. Twilio's server-side AEC is the common managed option for telephony, but it fails on cheap speakerphones and poor room acoustics. For high-stakes telephony deployments (healthcare, financial services), dedicated telephony AEC hardware or a service like Krisp's noise cancellation layer is worth evaluating. The rule is simple: confirm AEC is working before you start tuning VAD thresholds. A 30-second recording of the agent talking to itself is diagnostic. ## What breaks ### Mid-pause false cuts The single most-cited UX complaint. A user says: "I'd like to dispute a charge on my account — let me look up the amount." The pause after "account" is 400ms. Energy VAD at 300ms fires. The agent asks "what charge?" before the user has finished their context. The user restarts from scratch, the agent re-asks the question, and the experience degrades into mutual confusion. Mitigation: semantic VAD, or a longer energy threshold (500ms+). Longer threshold means slower response on clear short questions, so you're trading one bad experience for a slightly worse version of another. Semantic VAD is the better answer when your latency budget allows the extra 100ms. ### Barge-in storms In a noisy environment — a call center floor, a car on a highway — intermittent background noise triggers repeated speech-started events even when the user isn't talking. Each event fires the barge-in cancellation cycle. The agent never gets more than a few seconds into a response before being canceled. The call goes nowhere. Mitigation: noise suppression (RNNoise, Krisp, or WebRTC's built-in NS) upstream of the VAD, plus a minimum speech duration threshold (100ms minimum keeps random pops from triggering full cancellations). ### Double-talk and overlapping speech When user and agent genuinely speak simultaneously — not barge-in, but actual simultaneous overlapping speech — transcription quality falls off a cliff. Interspeech research measured 16.8% WER on clean single-speaker audio versus 74.6% WER on overlapping speech. That's not a marginal degradation; it means roughly three-quarters of overlapping words are wrong. Production pipelines handle this by serializing: when barge-in fires, stop the agent and process the user's next turn after. This produces correct-but-unnatural conversation that feels like a protocol rather than a dialogue. True full-duplex voice is an open research problem. ### TTS buffer drain timing A less obvious failure mode: the TTS synthesis pipeline runs ahead of playback. When barge-in fires, if you drain the buffer but don't abort in-progress TTS synthesis, the synthesis worker continues filling the buffer and the old content bleeds back in. Make sure your cancellation propagates all the way up the TTS call stack, not just to the buffer reader. ### Silent users and infinite wait Energy VAD waits for silence. If the user is thinking and stays silent for several seconds, the VAD either fires prematurely (short threshold) or never fires at all until you add a maximum silence timeout. Set a maximum turn silence ceiling — typically 10–15 seconds — after which the agent prompts ("Are you still there?"). Without this, a user who puts the call on hold can leave the pipeline hanging indefinitely. ``` Worked latency budget: energy vs semantic VAD Session: browser client, Deepgram Nova-3 STT, gpt-4o-mini LLM, Cartesia Sonic 4 TTS Energy VAD (200ms holdout): STT latency from speech end: 150ms (Deepgram streaming, flush on end-of-turn) LLM TTFT: 220ms TTS TTFB: 40ms (Cartesia Sonic 4) Network RTT: 80ms (US East, co-located) Total TTFA from speech end: ~490ms ✓ (sub-500ms target) False positive rate on real conversations: ~12% (measured on 200 calls) Semantic VAD (100ms model overhead): VAD overhead: +100ms STT latency from speech end: 150ms LLM TTFT: 220ms TTS TTFB: 40ms Network RTT: 80ms Total TTFA from speech end: ~590ms ✗ (just over 500ms target) False positive rate: ~1.5% Decision: the 100ms overage is acceptable given the 8× reduction in false positives. Users perceive 590ms as slightly slow but tolerable; they perceive being cut off as hostile. ``` The math shows why this is a genuine tension. Semantic VAD produces meaningfully better conversations but pushes past the 500ms perception threshold that separates "natural" from "slightly slow." For a voice agent where turn quality matters more than raw latency (support escalation, healthcare triage), semantic is the right call. For a high-volume outbound sales dialer optimizing for rapid yes/no responses, energy VAD at a short holdout might win. ## VAD in native speech-to-speech pipelines When you use a native S2S model — OpenAI's `gpt-realtime-1.5`, Gemini 3.1 Flash Live, or Amazon Nova Sonic — the turn detection is internal to the model. You don't wire up a separate VAD component; instead you configure turn-detection behavior on the session and respond to the events the API emits. This simplifies the architecture but reduces control. You can't swap in Silero VAD tuned to your noise floor, you can't run your own AEC preprocessing (you're sending raw audio to the API), and the debug story for "why did the agent cut me off?" is opaque. Native S2S models also cost significantly more — OpenAI Realtime at ~$0.30/min versus a cascaded pipeline with Deepgram + gpt-4o-mini + Cartesia at roughly $0.04–0.08/min — so every false barge-in is more expensive to produce. For the full comparison between cascaded and native architectures, see the sibling article on [cascaded vs native speech-to-speech](/ai-engineering/articles/cascaded-vs-native-speech-to-speech). ## Tuning in practice The right workflow is empirical, not theoretical: 1. **Record 100+ real user conversations** from a pilot or beta. Include your actual noise environment (office, car, call center). 2. **Measure false-positive barge-in rate**: count events where the VAD fired during a mid-sentence pause and the user hadn't finished. Target under 5%. 3. **Measure missed barge-in rate**: count events where the user tried to interrupt and the pipeline didn't respond within 300ms. Target under 10%. 4. **Tune threshold on this corpus**, not the vendor's clean-studio defaults. 5. **Re-measure after any infrastructure change** — codec changes, network path changes, and hardware changes all shift the noise floor. Deepgram and AssemblyAI both offer streaming VAD tuning parameters (`utterance_end_ms` in Deepgram, `end_utterance_silence_threshold` in AssemblyAI). Treat these as starting points. The vendor's defaults are tuned for their benchmark test suite, which does not resemble your production audio profile. Independent benchmarks from [Coval](https://www.coval.ai/blog/best-speech-to-text-providers-in-2026-independent-benchmarks-and-how-to-choose/) show 15–30% WER degradation in real call-center noise compared to vendor benchmarks — and VAD tuning degrades proportionally. ```mermaid flowchart TD START[Deploy with vendor defaults] --> RECORD[Record 100+ real calls] RECORD --> MEASURE[Measure false-positive rate] MEASURE --> HIGH{">5% false positives?"} HIGH -->|yes| LONGER[Increase silence threshold\nor switch to semantic VAD] HIGH -->|no| SLOW{">10% missed barge-ins?"} SLOW -->|yes| SHORTER[Decrease threshold\nor increase eagerness] SLOW -->|no| DONE[Ship — re-measure after\nany infra change] LONGER --> MEASURE SHORTER --> MEASURE style HIGH fill:#00e5ff,color:#0a0a0f style SLOW fill:#ffaa00,color:#0a0a0f style DONE fill:#15803d,color:#fff ``` ## When VAD is not your problem If users complain about the agent being slow to respond, that's almost certainly latency budget, not VAD. See [latency budget engineering](/ai-engineering/articles/voice-ai-latency-budget-production) for the full breakdown. If users complain about the agent ignoring questions or answering the wrong thing, that's more likely STT accuracy or LLM context management, not turn detection. VAD firing at the right time doesn't help if the transcript is wrong or the context window is full of stale history. If users complain that the agent doesn't understand their accent, that's STT model selection. Deepgram Nova-3 and AssemblyAI Universal-2 handle accented speech better than Whisper; the [voice pipeline overview](/ai-engineering/articles/voice-pipeline-stt-llm-tts) covers the full provider comparison. VAD is specifically the problem when users describe being "cut off," when the agent "doesn't let me finish," or when the agent "talks over itself." Those symptom descriptions are the diagnostic signal. ## The decision in practice Use **energy-based VAD** when: - Your latency budget is under 500ms and every millisecond counts (outbound dialer, rapid confirmation flows) - Your audio is clean and your users are concise (kiosk, IVR replacement) - You're using native S2S anyway and the model handles turn detection internally - Your volume is high and you're paying per minute on a managed platform — false positives cost money Use **semantic VAD** when: - Your use case involves complex user instructions or hesitant speech (support, healthcare, financial advice) - False-positive barge-ins are your top complaint and you have room to absorb ~100ms additional latency - You're on OpenAI Realtime and can set `turn_detection.type: "semantic_vad"` on the session - You have heard "it keeps cutting me off" from more than 5% of users in beta Side by side, using the numbers from the worked budget above (illustrative, mid-2026 pricing): | | Silence-based VAD (Silero, WebRTC, `server_vad`) | Semantic VAD | Native S2S internal (Realtime, Gemini Live) | |---|---|---|---| | End-of-turn latency overhead | ~0ms beyond the holdout itself | ~100ms turn-prediction model | Opaque — vendor-tuned, not separately measurable | | Mid-pause false-positive rate | ~12% untuned on real calls | ~1.5% | Generally low, but you can't instrument it | | Tunability | Full: threshold, holdout, min-speech duration | Coarse: an `eagerness`-style dial | Session config only | | AEC / preprocessing control | Full — your pipeline, your noise suppression | Full upstream of the turn model | None — raw audio goes to the API | | Cost of each false barge-in | One wasted STT flush + LLM call (~fractions of a cent) | Rare; same waste when it fires | Highest — ~$0.30/min platform pricing vs ~$0.04–0.08/min cascaded | The honest assessment: semantic VAD should be the default for almost all conversational agents. The 100ms additional latency is a fair price for not interrupting users mid-thought, and it's recoverable through other optimizations — streaming LLM tokens into TTS, regional inference routing, prompt prefix caching. Being cut off mid-sentence is not recoverable through anything except not doing it. The voice pipeline frameworks ([Vapi, LiveKit, Pipecat](/ai-engineering/articles/voice-agent-frameworks-vapi-livekit-pipecat)) all expose VAD configuration at different levels of granularity. Vapi abstracts most of it. LiveKit Agents and Pipecat give you direct access to the parameters. If you're at the stage where VAD tuning matters, you're probably past the point where Vapi's defaults are sufficient anyway. Turn detection is not a solved problem, and the field has not produced a universally correct answer. The 100ms semantic overhead will shrink as models get faster. Double-talk handling remains an open research area. What is solved: the causes of the most common failures, the parameters that control them, and the measurement approach that tells you whether your configuration is working. Start there. ## Frequently asked questions Q: What is voice activity detection (VAD) in AI voice agents? A: VAD is the component that decides when the user has started and finished speaking. Energy-based VAD fires on silence thresholds — typically 200–500ms of audio below a decibel floor. Semantic VAD uses a small language model to wait until the utterance is semantically complete before signaling end-of-turn. The distinction matters because energy-based VAD cuts users off mid-pause, the single most-cited UX complaint in production voice deployments. Q: What is the difference between energy-based VAD and semantic VAD? A: Energy-based VAD draws an end-of-turn boundary the moment audio drops below a silence threshold, typically 200–500ms. It is fast (~0ms overhead) but triggers on mid-sentence pauses, filler words, and hesitations. Semantic VAD runs a model that predicts whether the utterance is linguistically complete before ending the turn. It adds roughly 100ms of latency but dramatically reduces false-positive barge-ins. OpenAI added Semantic VAD to the Realtime API in 2025; LiveKit Agents 1.5.x (April 2026) ships adaptive interruption handling that adjusts sensitivity based on conversation history. Q: What happens during a barge-in event in a voice pipeline? A: When the user speaks while the agent is playing audio, the VAD fires a speech-started event. The pipeline must: (1) cancel the in-flight LLM generation, (2) discard any buffered TTS audio not yet played, (3) stop the audio output stream, and (4) begin transcribing the new user utterance as a fresh turn. Each of these steps must happen within a few hundred milliseconds or the user hears a jarring overlap. Failing to cancel the LLM call also wastes tokens and keeps goroutines pinned. Q: How bad is double-talk for voice AI accuracy? A: Severely. Interspeech research found word error rate rises from ~16.8% on clean single-speaker audio to 74.6% on overlapping speech. Most production pipelines do not handle true double-talk — they queue the user speech and process it serially, producing a walkie-talkie effect. Acoustic echo cancellation (AEC) removes the agent output from the microphone feed, but it does not reconstruct full-duplex simultaneous speech; it just prevents the agent from transcribing its own voice. Q: What silence threshold should I use for energy-based VAD? A: There is no universal answer, but common production starting points are 200–400ms of silence at –40 to –50 dBFS. The right value depends on your user population (native vs non-native speakers pause differently), channel quality (call-center noise floors differ from browser mic), and how fast you need the agent to respond. Tune by recording real conversations and measuring false-positive turn-end rate — aim for under 5% mid-sentence cuts. Silero VAD, the most widely deployed open-source option, gives you 30ms chunk-level probabilities that you can threshold. Q: When should I use native speech-to-speech models instead of cascaded VAD? A: Native S2S models like OpenAI's gpt-realtime-1.5 or Gemini 3.1 Flash Live handle turn detection internally in the audio stream — you don't wire up a separate VAD component. The tradeoff: you lose the intermediate transcript needed for tool calls, compliance logging, and CRM writes, and PSTN telephony (8 kHz audio) degrades native model quality while retaining premium pricing (~$0.30/min for OpenAI Realtime vs ~$0.02/min for Amazon Nova Sonic). See the sibling article on cascaded vs native speech-to-speech for the full decision framework.