--- schema: 1 type: video title: >- Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI description: >- Users notice latency above 500ms and hang up above one second. In an already optimized pipeline, 75ms of network latency from models sitting in a different data center adds 30% overhead. Colocating everything in the same building drops that to around 5ms. Rishabh Bhargava from Together AI walks through the full speech to text, LLM, and text to speech pipeline at that level of specificity. The LLM dominates the budget: 200 to 300ms time to first token target, 8 to 30B parameter range — larger models blow the latency budget, smaller ones break tool calling. Speech to text target is P90 under 100ms with around 6% word error rate. One pattern for handling complex workflows without adding latency: a small thinker LLM handles conversation flow and issues a single tool call to a larger model when the request is complex, keeping the fast path fast. Speaker info: - https://www.linkedin.com/in/bhargavarishabh --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/N7b1PJc7SFc-engineering-voice-agents-latency-quality-and-scale Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/N7b1PJc7SFc' markdown: 'https://liquidgold.cc/watch/N7b1PJc7SFc.md' llms: 'https://liquidgold.cc/watch/N7b1PJc7SFc/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T10:07:40.808Z' language: null children: [] videoId: N7b1PJc7SFc durationSeconds: 1475 publishedAt: '2026-05-31T16:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 136 endSeconds: 238 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=136' heldReason: rights - ordinal: 1 text: null startSeconds: 238 endSeconds: 249 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=238' heldReason: rights - ordinal: 2 text: null startSeconds: 249 endSeconds: 329 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=249' heldReason: rights - ordinal: 3 text: null startSeconds: 329 endSeconds: 419 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=329' heldReason: rights - ordinal: 4 text: null startSeconds: 329 endSeconds: 419 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=329' heldReason: rights - ordinal: 5 text: null startSeconds: 419 endSeconds: 539 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=419' heldReason: rights - ordinal: 6 text: null startSeconds: 419 endSeconds: 539 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=419' heldReason: rights - ordinal: 7 text: null startSeconds: 539 endSeconds: 660 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=539' heldReason: rights - ordinal: 8 text: null startSeconds: 539 endSeconds: 660 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=539' heldReason: rights - ordinal: 9 text: null startSeconds: 539 endSeconds: 660 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=539' heldReason: rights - ordinal: 10 text: null startSeconds: 660 endSeconds: 759 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=660' heldReason: rights - ordinal: 11 text: null startSeconds: 660 endSeconds: 759 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=660' heldReason: rights - ordinal: 12 text: null startSeconds: 759 endSeconds: 866 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=759' heldReason: rights - ordinal: 13 text: null startSeconds: 866 endSeconds: 966 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=866' heldReason: rights - ordinal: 14 text: null startSeconds: 866 endSeconds: 966 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=866' heldReason: rights - ordinal: 15 text: null startSeconds: 966 endSeconds: 1094 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=966' heldReason: rights - ordinal: 16 text: null startSeconds: 966 endSeconds: 1094 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=966' heldReason: rights - ordinal: 17 text: null startSeconds: 1184 endSeconds: 1317 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=1184' heldReason: rights - ordinal: 18 text: null startSeconds: 1184 endSeconds: 1317 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=1184' heldReason: rights - ordinal: 19 text: null startSeconds: 1317 endSeconds: 1475 watch: 'https://liquidgold.cc/watch/N7b1PJc7SFc?t=1317' heldReason: rights --- # Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI Users notice latency above 500ms and hang up above one second. In an already optimized pipeline, 75ms of network latency from models sitting in a different data center adds 30% overhead. Colocating everything in the same building drops that to around 5ms. Rishabh Bhargava from Together AI walks through the full speech to text, LLM, and text to speech pipeline at that level of specificity. The LLM dominates the budget: 200 to 300ms time to first token target, 8 to 30B parameter range — larger models blow the latency budget, smaller ones break tool calling. Speech to text target is P90 under 100ms with around 6% word error rate. One pattern for handling complex workflows without adding latency: a small thinker LLM handles conversation flow and issues a single tool call to a larger model when the request is complex, keeping the fast path fast. Speaker info: - https://www.linkedin.com/in/bhargavarishabh --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/N7b1PJc7SFc-engineering-voice-agents-latency-quality-and-scale Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/N7b1PJc7SFc · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.