The 500-millisecond wall: why voice AI latency is harder than it looks

In February 2026, developer Nick Tikhonov published “I built a sub-500ms latency voice agent from scratch”, and it landed hard on Hacker News — 570 points and 153 comments arguing over what “real-time” actually means for a voice agent. His framing of the problem is the useful part: “Voice agents are a big step-change in complexity compared to agentic chat,” he writes, because “real-time systems are unforgiving, and humans are extremely sensitive to timing errors.” His numbers back that up — his own pipeline ran ~1.6-1.7s locally, dropped to ~690-790ms on an EU deployment, and hit roughly 400ms end-to-end once he swapped in Groq’s inference for the LLM step, with first-token time alone eating “more than half of the total latency.” His conclusion on architecture is blunt: “A production voice agent cannot be built as STT → LLM → TTS as three sequential steps” — it has to be a streaming pipeline with immediate cancellation on interruption, not three services bolted together.

The comment section turned into a genuinely useful reality check on what “good” latency even means. Commenter jedberg pointed out that the incumbents aren’t fast either — “Almost no response from Alexa is under 500ms. Even the ones that are served locally, like ‘what time is it’” — and then raised the actual human bar: “The median delay between human speakers during a conversation is 0ms (zero). In many cases, the listener starts speaking before the speaker is done.” Commenter stonelazy pushed back on comparing a demo to production: “Your system is a clean straight pipe: transcript -> LLM -> TTS -> audio. No tool calls, no function execution, no webhooks… Production platforms like Vapi are doing way more work on every single turn.” And a couple of commenters flagged the failure mode on the other side of tuning latency too aggressively — user gytdev: “I hate that LLM’s interrupt me when talking even though I haven’t finished my thought and was just thinking quite slow,” and ismailmaj on the tuning trade-off: “Semantic end of turn being 300ms of silence is horrible because I ended up intentionally um-ing to finish my thoughts.”

That thread is a good summary of where voice AI actually is in mid-2026: raw latency numbers are becoming commoditized talking points, but the harder, less-benchmarked problem is turn-taking — knowing when a person is actually done speaking, not just when they’ve paused.

What this means for buyers

If you’re evaluating a voice AI vendor, a headline latency number without the interruption-handling and production-load context behind it (tool calls, function execution, concurrent sessions) tells you less than it looks like it does — ask what the number was measured against, per stonelazy’s point above. It’s also worth noting that the category has real vendor churn to account for: Meta acquired PlayHT (rebranded Play AI) in 2025, and the service was shut down by early 2026 — a live reminder to verify a voice AI vendor is still operating before committing to it, not just still ranking well in search. We track current status on all of these on Crail’s voice & speech AI category page, and compare two of the leading players in ElevenLabs vs. Deepgram.