--- schema: 1 type: video title: '5 Voice Agent Failure Modes You''ll Hit in Week One — Venky B, Plivo' description: >- Nearly every intelligence gain in language models over the past year has come from letting them think longer. Voice agents have to turn thinking off, because the budget between a user finishing a sentence and the agent starting to speak is measured in hundreds of milliseconds. Venky B is founder and CEO of Plivo, which carries over a billion voice calls a month and has been building telephony infrastructure since 2011, and this talk is a tour of what breaks when a voice agent leaves the demo and meets production. On latency his numbers are blunt. Teams aim for under 550 milliseconds and most land between 750 and 1,200, and past that users simply hang up. His team's answer is smaller open source models hosted themselves, targeting under 300 milliseconds, chosen partly on how many tokens a language needs per word. The failure he says wrecks half of all deployments is data collection, and his fix is to stop treating it as transcription at all. Decide the shape before you ask. A phone number is a typed field with a length and a validator, so a stray letter in the middle is either corrected with confidence or sent back to the caller, and evaluation happens per field as a unit test rather than end to end. That reframing took his accuracy from roughly 30 percent to the mid nineties with no fine tuning. He is equally specific about transcription being brittle by default, especially with proper nouns and code switched languages, and about never feeding model output straight into speech synthesis. His own benchmark for a vendor is whether it can pronounce his surname and his company's name. Speaker info: - https://x.com/bevenky - https://www.linkedin.com/in/bevenky/ Timestamps: 0:00 - Where voice agents break between demo and production 2:20 - A billion calls a month 4:28 - The pipeline everyone builds first 5:34 - Failure one: latency and time to first audio 6:42 - Balancing cost, intelligence, and latency 7:48 - Why thinking models do not fit 9:56 - Choosing and sizing open source models 13:06 - Failure two: assume transcription is brittle 15:14 - Code switched languages break everything downstream 16:17 - Dynamic keyword boosting and LLM post processing 18:24 - Failure three: collect data as typed fields 20:31 - Relative dates and other traps 21:37 - Field level evals instead of end to end 22:47 - Failure four: normalize before synthesis 25:00 - Failure five: turn detection and barge in --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/vblnYHzBgS4-5-voice-agent-failure-modes-youll-hit Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/vblnYHzBgS4' markdown: 'https://liquidgold.cc/watch/vblnYHzBgS4.md' llms: 'https://liquidgold.cc/watch/vblnYHzBgS4/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T03:35:55.132Z' language: null children: [] videoId: vblnYHzBgS4 durationSeconds: 1606 publishedAt: '2026-09-15T15:30:09.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 288 endSeconds: 334 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=288' heldReason: rights - ordinal: 1 text: null startSeconds: 347 endSeconds: 407 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=347' heldReason: rights - ordinal: 2 text: null startSeconds: 407 endSeconds: 420 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=407' heldReason: rights - ordinal: 3 text: null startSeconds: 420 endSeconds: 477 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=420' heldReason: rights - ordinal: 4 text: null startSeconds: 489 endSeconds: 604 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=489' heldReason: rights - ordinal: 5 text: null startSeconds: 489 endSeconds: 604 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=489' heldReason: rights - ordinal: 6 text: null startSeconds: 604 endSeconds: 615 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=604' heldReason: rights - ordinal: 7 text: null startSeconds: 615 endSeconds: 788 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=615' heldReason: rights - ordinal: 8 text: null startSeconds: 615 endSeconds: 788 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=615' heldReason: rights - ordinal: 9 text: null startSeconds: 800 endSeconds: 921 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=800' heldReason: rights - ordinal: 10 text: null startSeconds: 800 endSeconds: 921 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=800' heldReason: rights - ordinal: 11 text: null startSeconds: 921 endSeconds: 932 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=921' heldReason: rights - ordinal: 12 text: null startSeconds: 986 endSeconds: 998 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=986' heldReason: rights - ordinal: 13 text: null startSeconds: 998 endSeconds: 1109 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=998' heldReason: rights - ordinal: 14 text: null startSeconds: 1109 endSeconds: 1121 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=1109' heldReason: rights - ordinal: 15 text: null startSeconds: 1121 endSeconds: 1233 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=1121' heldReason: rights - ordinal: 16 text: null startSeconds: 1233 endSeconds: 1244 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=1233' heldReason: rights - ordinal: 17 text: null startSeconds: 1301 endSeconds: 1312 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=1301' heldReason: rights - ordinal: 18 text: null startSeconds: 1369 endSeconds: 1380 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=1369' heldReason: rights - ordinal: 19 text: null startSeconds: 1380 endSeconds: 1506 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=1380' heldReason: rights - ordinal: 20 text: null startSeconds: 1517 endSeconds: 1606 watch: 'https://liquidgold.cc/watch/vblnYHzBgS4?t=1517' heldReason: rights --- # 5 Voice Agent Failure Modes You'll Hit in Week One — Venky B, Plivo Nearly every intelligence gain in language models over the past year has come from letting them think longer. Voice agents have to turn thinking off, because the budget between a user finishing a sentence and the agent starting to speak is measured in hundreds of milliseconds. Venky B is founder and CEO of Plivo, which carries over a billion voice calls a month and has been building telephony infrastructure since 2011, and this talk is a tour of what breaks when a voice agent leaves the demo and meets production. On latency his numbers are blunt. Teams aim for under 550 milliseconds and most land between 750 and 1,200, and past that users simply hang up. His team's answer is smaller open source models hosted themselves, targeting under 300 milliseconds, chosen partly on how many tokens a language needs per word. The failure he says wrecks half of all deployments is data collection, and his fix is to stop treating it as transcription at all. Decide the shape before you ask. A phone number is a typed field with a length and a validator, so a stray letter in the middle is either corrected with confidence or sent back to the caller, and evaluation happens per field as a unit test rather than end to end. That reframing took his accuracy from roughly 30 percent to the mid nineties with no fine tuning. He is equally specific about transcription being brittle by default, especially with proper nouns and code switched languages, and about never feeding model output straight into speech synthesis. His own benchmark for a vendor is whether it can pronounce his surname and his company's name. Speaker info: - https://x.com/bevenky - https://www.linkedin.com/in/bevenky/ Timestamps: 0:00 - Where voice agents break between demo and production 2:20 - A billion calls a month 4:28 - The pipeline everyone builds first 5:34 - Failure one: latency and time to first audio 6:42 - Balancing cost, intelligence, and latency 7:48 - Why thinking models do not fit 9:56 - Choosing and sizing open source models 13:06 - Failure two: assume transcription is brittle 15:14 - Code switched languages break everything downstream 16:17 - Dynamic keyword boosting and LLM post processing 18:24 - Failure three: collect data as typed fields 20:31 - Relative dates and other traps 21:37 - Field level evals instead of end to end 22:47 - Failure four: normalize before synthesis 25:00 - Failure five: turn detection and barge in --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/vblnYHzBgS4-5-voice-agent-failure-modes-youll-hit Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/vblnYHzBgS4 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.