#latency
14 articles
Voice AI latency budget: hitting sub-500ms in production
A systematic breakdown of every millisecond in the voice AI pipeline and the specific techniques that compound to sub-500ms time-to-first-audio in production.
Voice Agent Frameworks: Vapi vs LiveKit vs Pipecat
Compare Vapi, LiveKit Agents, and Pipecat across latency, cost, telephony, and control — with the exact break-even math to pick the right abstraction.
Streaming LLM Responses: The Engineering Nobody Talks About
SSE vs WebSockets, partial-JSON parsing for streamed tool calls, backpressure, cancellation, and rendering streamed markdown without flicker — the complete engineering guide.
VAD and Turn Detection: The Hardest UX Problem in Voice AI
How voice activity detection and turn-taking work in real-time AI agents — energy vs semantic VAD, barge-in handling, and why getting this wrong is the top user complaint.
Throughput vs Latency: The GPU Economics of LLM Serving
How tokens/sec, cost per token, TTFT SLAs, and GPU utilization interact — and the engineering decisions that separate a profitable inference deployment from a money pit.
Cascaded pipeline vs native speech-to-speech: when to use each
Chain STT, LLM, and TTS for control — or use one speech-native model for latency. The real tradeoffs, with numbers, and when each architecture wins.
The Voice Pipeline: STT to LLM to TTS End-to-End
Build the STT to LLM to TTS voice pipeline end-to-end — streaming each stage, budgeting latency, and hitting sub-second time-to-first-audio.
Speculative decoding: 2-3x latency speedup without changing model quality
How a small draft model proposes tokens the large target verifies in parallel — and why rejection sampling guarantees identical output distributions with real speedup numbers.
Streaming, Batching, and Throughput: Latency vs Cost Trade-offs in Production
How streaming, continuous batching, and async batch APIs interact with latency and cost — and how to configure each without destroying p99 for interactive users.
Cost, latency, and the model-selection decision tree for reasoning
A practical framework for choosing between reasoning and standard models, with real cost and latency numbers from 2025-2026 deployments.
Model routing and cascades: sending the right query to the right model
How to build rule-based routers, intent classifiers, and confidence-based cascades that cut your LLM API bill 60–85% while preserving 95% of frontier quality.
Continuous Batching and Scheduling: Why Static Batching Wastes 70% of Your GPU
How iteration-level scheduling replaced static batching and became the single biggest serving optimization available without any model changes — with real numbers.
Prompt caching deep dive: prefix caching, semantic caching, and when each wins
How provider-side prefix caching and application-side semantic caching work, what they cost, where each breaks, and the decision framework for combining them in production.
Prefill vs Decode: The Two Phases of Every LLM Request
Why the first token takes longer than every other token, how compute-bound prefill and bandwidth-bound decode shape your entire serving architecture, and what to do about it.