Rishabh Bhargava (Together AI) breaks down the engineering of voice agent pipelines — speech-to-text, LLM, and text-to-speech — with concrete latency budgets: users notice latency above 500ms and hang up above one second. He explains how 75ms of network latency from a non-colocated model adds ~30% overhead in an optimized pipeline, versus ~5ms when colocated. The LLM dominates the budget (200–300ms time-to-first-token target, 8–30B parameter range; larger models exceed the latency budget, smaller ones break tool calling), while speech-to-text targets P90 under 100ms at roughly 6% word error rate. He also covers a pattern for complex workflows: a small "thinker" LLM manages conversation flow and delegates to a larger model only when needed, keeping the fast path fast. # Video Identity Extraction: Observable Channel Presentation Evidence ## Video Overview - **Title:** *Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI* - **Format:** Conference/meetup talk with live audience Q&A, given by an industry practitioner. ## Explicit Self-Description (Creator Claims) - **Speaker role claim:** "I work at a company called Together AI. I lead the voice AI team there." - **Company self-description:** "Together is building the AI native cloud… for companies that are looking to train models and need access to reliable compute, or you want to do inference at scale." - **Scale claims:** "There's a million plus developers. We closely work with hundreds of companies… working with companies like Cursor and Deck." - **Personal background claim:** "Prior to Together, I was the co-founder CEO of a company called Refuel that was acquired by Together last year. But generally been building AI and machine learning infrastructure for about a decade." - **Recruitment signal:** "We are hiring. We also have a booth G1 downstairs." ## Intended Audience - **Engineers and technical builders** of voice AI systems — evidenced by deep technical content: latency budgets (TTFT, TTFA, real-time factor), word error rate, pipeline architecture, auto-scaling nuances. - **Practitioners making production decisions** — the talk frames choices in production terms: "the dominant way of building these voice agents… in production today," model sizing ("8 to 30 billion range"), deployment/residency considerations. - **Live conference attendees asking applied questions** — Q&A covers evals, guardrails, and co-location, indicating an audience actively building voice agents. ## Claimed Scope of Content The speaker states an explicit agenda: > "We're going to first talk about… why does voice really matter? What does it actually take to build voice agents at scale?… We'll talk about this pipeline architecture… We'll look at the system and trade-offs. And then finally… what might be the next generation of building voice agents." The talk delivers on this scope: market rationale, pipeline components (speech-to-text, LLM, text-to-speech), system concerns (latency/cost budgets, auto-scaling, global deployment, co-location), and speech-to-speech model futures. ## Demonstrated Presentation Patterns (Observable, Not Claimed) - **Vendor-adjacent but technically neutral framing:** Together AI is positioned in the intro/outro only; the body discusses competitors and third parties openly (OpenAI's real-time API, Nvidia's voice chat model, Whisper, Cursor, Claude Code) without promotional framing. - **Practitioner credibility via specifics:** Concrete numbers throughout — "P90 of like 100 milliseconds" for transcription, "300 milliseconds of TTFT," "6% word error rate," "75 milliseconds to five basically gets you a 30% reduction." - **Balanced/uncertain framing on open problems:** "Turn detection, very important, still somewhat unsolved problem," and honest assessment of speech-to-speech models: "the reason why most of these models are not used in production too much is because they… still have trouble with instruction following." - **Engagement with audience expertise:** Speaker validates audience questions ("Yeah, that's spot on"), acknowledges limits ("unfortunately, no easy answers"), and answers the classic "it depends" on evals. - **Hiring/booth call-to-action** closes the talk — observable channel behavior consistent with a company-affiliated speaker at an industry event. ## Identity Evidence Summary | Dimension | Evidence | |---|---| | Speaker | Rishabh Bhargava, leads voice AI at Together AI (self-stated) | | Affiliation | Together AI ("AI native cloud") — explicit self-description | | Audience | Voice agent engineers / AI infrastructure builders | | Scope | Engineering voice agents: latency, quality, scale — matched by delivered content | | Tone | Technical, production-focused, candid about trade-offs | No private motivations or channel-wide conclusions were inferred; all observations trace to quoted transcript passages. # Delivery Observations: Rishabh Bhargava — "Engineering voice agents: Latency, quality, and scale" ## Overall Tone and Register - **Conversational, informal engineering talk.** The speaker opens casually: "All right. Well, folks, thanks for being here. Excited to chat more about how to engineer voice agents." - Frequent use of **"you know"** as a filler/connective phrase, appearing dozens of times across the talk — a stable, defining verbal habit: > "Excited to chat more about how to engineer voice agents, you know, high quality, low latency at scale." > "There's a few things that you got to solve first." - Uses **direct second-person address** ("you got to get latency down," "you might actually end up having your models being sufficiently far away") to walk the audience through scenarios. ## Teaching Habits and Structural Patterns - **Explicit agenda-setting:** "Okay, here is the agenda for today. So, we're going to first talk about... why we're talking about voice." - **Numbered/list framing:** "First... The next thing that matters... The third piece that you got to solve for... And finally." Used both for why-voice-matters and component breakdowns. - **Section transitions with signposts:** "Okay, so that's speech to text. Jumping into the next part of the pipeline, which is LLMs." and "Okay, text to speech." - **Analogies for pipeline components:** > "very much like the ears of your agent" > "very much the brains of your agent" > "very much, you know, the voice of your agent" - **Concrete numeric anchors:** repeatedly cites specific figures — "300 milliseconds," "6% word error rate," "P90 of like 100 milliseconds," "8 to 30 billion range," "75 milliseconds to five basically gets you a 30% reduction." - **"Rough rubric" heuristics:** "A rough kind of, you know, metric is like... it's usually pretty good if you can get to 300 milliseconds of TTFT" — favors rules of thumb over exhaustive precision. ## Recurring Vocabulary and Phrases - **"Kind of" / "sort of"** as frequent hedges: "a rough architecture," "this is just kind of stepping out," "sort of like the pipeline architecture." - **"Frankly"** as an emphasis marker (multiple occurrences): > "But really one of the more exciting directions is, frankly, voice is just this brand new interface" > "This is primarily an engineering problem today." > "frankly nothing quite beats listening to audio samples" > "frankly, the tool calling that is needed" - Engineering vocabulary: latency, TTFT, P90, tool calling, guardrails ("God reals" — likely a transcription of "guardrails"), auto-scaling, co-location, observability, real-time factor. ## Audience Engagement Patterns - **Relatable hypotheticals** to ground technical points: > "I am pretty sure all of us have the experience of calling customer support... And pretty much everybody here probably has had the experience of being on hold." > "Look, humans, we learned how to talk before we learned how to read, right?" - **Rhetorical questions followed by self-answers**, especially in Q&A: > "So, in that what I understand... Yeah, good question. So, for example..." > "that's like the classic answer is like it depends" - **Openly acknowledges limits/uncertainty** rather than overclaiming: > "no easy answers" > "Turn detection, very important, still somewhat unsolved problem, frankly." > "as these models get better, which I'm confident they will" (a forecast, flagged as opinion) ## Q&A Delivery Style - Validates questioners before answering: "Yeah, it's a great question," "very good question," "I think that's spot on." - **Concrete hypothetical walkthroughs** as explanation method — e.g., the London/US OpenAI co-location example: "let's say you're building a voice agent here in London, right?" - Willing to defer detail: "That could be a 20-minute talk in itself." - When uncertain or case-dependent, says so directly: "again, it depends a little bit on what does the use case kind of eventually demand." ## Stability vs. Isolated Moments - **Stable patterns** (high repetition): "you know," "kind of/sort of," numbered framing, numeric anchors, second-person walkthroughs, agenda signposting. - **Isolated moments:** the "homi-like communication" phrasing (likely "human-like," [966]), the "I should have lied. You can't take back things that are spoken" exchange late in Q&A appears to be audience interplay rather than the speaker's prepared style — evidence is thin and the transcript is garbled there, so interpretation is uncertain. ## Uncertainty Notes - Transcript contains apparent speech-to-text artifacts ("God reals" for guardrails, "homi-like," "Deck again," "pipe cat") — quoted as-is; the intended words are plausible but not verifiable from this text alone. - Filler density ("you know") may be partly an artifact of the auto-transcription; the underlying frequency is still clearly high based on consistent distribution. # Position Record: Engineering Voice Agents — Latency, Quality, and Scale (Rishabh Bhargava, Together AI) ## Speaker - **Rishabh Bhargava**, lead of the voice AI team at **Together AI**; previously co-founder/CEO of Refuel (acquired by Together); ~decade building AI/ML infrastructure. ## Why Voice Matters - **Billions of phone calls** per year still handled by humans (customer support, reservations, appointments). - Voice is a **brand-new interface**: humans "learned how to talk before we learned how to read." - Cites ChatGPT advanced voice mode, talking to Cursor, and Claude Code as evidence this trend has begun. - **Claim:** Building rich, high-quality voice conversations "is not the domain of science fiction or research anymore" — it is "primarily an engineering problem today." ## Why Voice Is Hard (Four Simultaneous Requirements) - **Real-time latency:** Humans respond to conversational cues in ~**300 milliseconds**; responses over 500 ms are noticeable, and 1–2 second delays cause people to hang up. - **Baseline intelligence:** Complex real-world workflows require strong instruction-following and **tool calling** ("the way you give agents access to the real world"). - **Naturalness of voice:** Right language/accent, name pronunciation, emotional delivery. - **Reliability at scale:** A demo works with one call; 100–10,000 concurrent calls demand reliability. - **Framing:** "This is an and problem. You have to solve every single one of them at the same time." ## Dominant Architecture: Pipeline / Cascading Audio chunks → agent orchestrator (Pipecat, LiveKit, or homegrown) → **speech-to-text** → **LLM** (tool calling decisions) → **text-to-speech** → streamed audio back to user. ### Speech-to-Text ("the ears") - **Quality metric:** word error rate — state-of-the-art ~**6%** on open benchmarks. - Errors propagate downstream: "there's no way to fix this. Your LLM will carry forward the mistake. The TTS model will carry forward the mistake." - **Latency metric:** time to complete transcript after an utterance ends; Together-hosted models reach a consistent **P90 of ~100 ms**. - **Turn detection:** "still somewhat unsolved problem, frankly" — detecting whether a pause means the turn ended. - **Language coverage** matters depending on customers. - **Architectural shift — batch → streaming-native:** Whisper was trained on 30-second clips, requiring "complicated logic" (chunking, silence padding, stitching). A recent **NVIDIA model** uses an encoder trained with variable look-ahead (~80 ms to ~1 s) and activation caching so heavy computation happens once. ### LLM ("the brains") - Key metric: **TTFT**; ~**300 ms** is a good target so tokens feed the TTS quickly. - Budget constrains model size: typically **8–30 billion parameters**. Bigger burns latency budget; smaller hurts intelligence and tool calling. ### Text-to-Speech ("the voice") - **TTFA** (time to first audio) and **real-time factor (RTF)** — RTF < 1 to avoid buffering (e.g., 10 s of audio produced in 5 s = 0.5). - **Quality caveat (stated honestly):** objective measures exist, but "nothing quite beats listening to audio samples." - Capabilities: naturalness across voices, exact pronunciation (names, product names), **emotion control** via tags ("happy or angry or sad") — "it's the start, but these models are getting pretty good." - Language coverage continues to matter. ## System-Level Considerations - **Latency/cost budget split:** LLM takes the majority, then TTS, then STT. - **Network latency vs. engine latency:** Models in different data centers add hops; a 75 ms network latency example against 100–200 ms engine latency. - **Co-location argument:** Moving models into the same data center as the orchestrator (75 ms → ~5 ms) yields "a 30% reduction in already a fairly optimized voice agent setup." In Q&A, he elaborates: e.g., a London-built agent using OpenAI US-hosted models pays the transatlantic hop; running an open-source model in the same data center avoids it. "Distance literally... has that big of an impact." - **Auto-scaling nuance:** Scale up aggressively ("you typically might auto scale earlier than... asynchronous systems"); scale down is tricky due to **stateful long-lived connections** — "you can't just arbitrarily kill a pod." - **Global deployment** for latency and data residency. - **Observability:** "Every 10 milliseconds matters." ## Alternative: Pure Speech-to-Speech Models - **Claimed benefit:** Simpler — one model handling function calling and complex instructions; OpenAI's real-time API and NVIDIA's "voice chat" model follow this idea. - **Stated limitation (preserved as-is):** "The reason why most of these models are not used in production too much is because they still have trouble with instruction following and tool calling." Typical experience: prompt-engineering attempts, then falling back to a pipeline. - **Expressed confidence:** "As these models get better, which I'm confident they will..." benefits include preserving tone/emotion/hesitation lost in text conversion, **full-duplex communication** with back-channeling ("I see," "aha"), and better interruption/barge-in handling. - Overall stance: "We have a lot of engineering work ahead" for these interfaces. ## Q&A Positions ### Evals for voice-to-function-calling - "Classic answer is like it depends" (his own framing). - Component-by-component evals; assuming STT/TTS are good, LLM tool-calling evals resemble general LLM evals: was the tool call correct? Was the output possible? - Target: tool call structure "at least very close to 100%"; correctness depends on the use case. - **Observed pattern:** Customers **fine-tune smaller LLMs** with use-case-specific data to raise tool-calling quality while staying within the latency-budgeted small-model size. ### Guardrails/classifiers in the pipeline - The reference architecture is "the most basic"; production systems add **classifiers before the LLM** (routing refunds vs. order tracking) and **guardrails after LLM generation** before TTS. - Trade-off acknowledged: added components "put real pressure on latency" — "unfortunately, no easy answers," beyond clear guidelines, SLAs, and independent scaling. - **Questioner's unresolved concern (preserved):** spoken errors cannot be taken back; you might have to say "sorry, I shouldn't have said that. I need to revoke that." - Bhargava agrees catching errors **before TTS invocation** is important. - **"Thinker-talker" pattern:** a small LLM handles conversation and produces stalling responses ("let me think about it"), while issuing tool calls to a **much bigger model** with better instructions and guardrails, whose cleaner output feeds TTS. ### Evals/observability for speech-to-speech models - Some observability can remain: run a parallel **transcription model** for auditability. - Evals will change: no pure text-to-text or text-to-speech evals; evals become **full-duplex, whole-conversation metrics**. - **Final answer (questioner pressed on whether models natively expose evaluable characteristics):** "Typically a lot of the evals that would happen on top of the base inference API." Session ended shortly after; no further elaboration. ## Preserved Tensions & Qualifications - Pipeline is "dominant today" but speech-to-speech is "becoming interesting" — the speaker presents both without declaring a winner. - STT turn detection: "still somewhat unsolved." - TTS quality: subjective listening remains the gold standard despite objective measures. - Guardrail latency problem: explicitly "no easy answers." - Speaker self-describes Together as "probably a very good fit" for training/inference compute (promotional context acknowledged in the record). # Observation Document: Speaker Roles & Provenance ## Video - **Title:** Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI - **Video ID:** `01a15ca9-17f3-783e-b28b-13a286fbb43f` - **Format:** Conference-style talk with audience Q&A ## Speakers & Roles ### Rishabh Bhargava — **Creator / Primary Speaker** - Self-identifies as "Rishab," leading the **voice AI team at Together AI**. - Prior background (self-reported): **co-founder/CEO of Refuel**, acquired by Together; ~decade building AI/ML infrastructure. - Delivers the full presentation and answers all audience questions. ### Unidentified Audience Members — **Guests (Q&A participants)** - Multiple unnamed questioners during Q&A; questions on: - Evals and production score thresholds for voice-to-function-calling. - Co-location and network latency implications with cloud providers. - Guardrail/classifier models in the pipeline without compromising latency. - Observability/evals for upcoming speech-to-speech models. - One audience member references their own experience ("when trying this voice pipeline system that we had to introduce some guard railing... 8% discount"). ### Referenced Prior Speaker - A **"previous speaker"** is alluded to (working on voice); unnamed, no content attributed. ## Referenced Sources & Third-Party Claims - **Products/companies cited by the speaker:** - **Together AI** — "AI native cloud"; claims of 1M+ developers, work with "hundreds of companies," naming **Cursor** and "Deck" (transcript garbled) as customers. - **Whisper** — described as canonical batch speech-to-text model (30-second clips). - **Nvidia** — cited for a new streaming speech-to-text model and "voice chat" model. - **OpenAI Real-Time API** — cited as single speech-to-speech model; ChatGPT advanced voice mode mentioned. - **PipeCat / LiveKit** — named as agent orchestrator examples. - **Technical claims are speaker-attributed** (e.g., ~300ms human turn-taking response, 6% WER on "open benchmarks," P90 ~100ms on Together, 75ms network latency example). No external citations provided in the transcript. ## Promotional Segments (Flagged) - **Opening self/company introduction:** Together AI positioning, developer/customer counts, named customers. - **Closing segment:** "if you want to learn more about what Together does... here are a couple of links. **We are hiring. We also have a booth G1 downstairs.**" — explicit promotion. ## Availability & Limitations - **No curated metadata supplied** — no speaker bios, cited-links list, comment availability, or timestamps beyond the raw transcript. - Q&A questioners are **unidentified by name or affiliation**; boundaries between questioners are not marked. - Some transcript garbling present (e.g., "Rishab" vs. title "Rishabh"; "God reals" for "guardrails"; "Deck again"; "homi-like"). All right. Well, folks, thanks for being here. Excited to chat more about how to engineer voice agents, you know, high quality, low latency at scale. First, maybe a little bit about me. My name is Rishab. I work at a company called Together AI. I lead the voice AI team there. Prior to Together, I was the co-founder CEO of a company called Refuel that was acquired by Together last year. But generally been building AI and machine learning infrastructure for about a decade. For folks who maybe don't know about Together, Together is building the AI native cloud. What that really means is for companies that are looking to train models and need access to reliable compute, or you want to do inference at scale, we're probably a very good fit for you. We work with, you know, there's a million plus developers. We closely work with hundreds of companies and, you know, very proud to be working with companies like Cursor and Deck again. Okay, here is the agenda for today. So, we're going to first talk about, you know, why we're talking about voice, although the previous speaker alluded to a lot of interesting things that he's doing with voice. But why does voice really matter? What does it actually take to build voice agents at scale? What are the challenges? We'll talk about this pipeline architecture, which is becoming the dominant way to build agents in production today. We'll go a little bit deep dive a little bit. We'll look at all the components. We'll look at the system and trade-offs. And then finally, we'll chat about maybe what might be the next generation of building voice agents in the coming months and years. Okay, so starting with why voice matters. Well, there's billions of phone calls a year that are still handled by humans. I am pretty sure all of us have the experience of calling customer support, asking about the status of our order, looking to change the reservation. We've also probably had the experience of calling a doctor's office because we got to book an appointment for ourself or a loved one. And pretty much everybody here probably has had the experience of being on hold. Now, it would be amazing for AI agents to be able to handle some of these calls for us. But really one of the more exciting directions is, frankly, voice is just this brand new interface to interact with systems and computers. Look, humans, we learned how to talk before we learned how to read, right? So, this comes very naturally to us. And obviously, we're seeing this with ChatGPT's advanced voice mode. And we're seeing this with folks who are starting to directly talk to Cursor, talk to Claude code in order to get their work done. And this is, frankly, just the beginning. And one of the exciting pieces of 2026 is, you know, building these rich, high-quality conversations. This is not the domain of science fiction or research anymore. This is primarily an engineering problem today. Now, why is it hard? Well, there's a few things that you got to solve first. First, building voice AI and building voice agents, this has to be real time. You know, when humans are having a conversation, we respond to each other's cues in something like 300 milliseconds. And so, if you're talking to an AI and it's taking more than 500 milliseconds to respond, you'll start to notice. If it takes a second, if it takes 2 seconds, people will just hang up. So, you got to get latency down. The next thing that matters is you want it to be a reasonably smart call. You want to get the work, the job done. And so, for real-world complex workflows, you know, the instructions are complicated. There's a lot of ambiguity. You have to be good at tool calling because that's the way you give agents access to the real world. So, you have a baseline level of intelligence that you got to meet. The third piece that you got to solve for is the voice has to be natural enough. It has to sound pleasant enough. And this means a lot of different things. It means, you know, can it talk to you in your own language with the right accent potentially? Can it pronounce your name? Can it deliver the right emotion that is needed for a particular situation? A lot of the things fall into that bucket. And finally, you know, you could switch together like a nice demo with one person calling. But what happens when you're doing 100 calls, 1,000 calls, 10,000 calls concurrently? Reliability really starts to matter. And this is an and problem. You have to solve every single one of them at the same time, or you're going to be in a little bit of a trouble. So, at least today, the dominant way of building these voice agents is this pipeline architecture or this cascading architecture, which attempts to solve all of the problems that I outlined earlier. Now, there's a few boxes that are going on, but, you know, it conceptually it's relatively simple to understand, which is audio chunks from an end user that are being streamed in, potentially to an agent orchestrator, something like a pipe cat or live kit or something that is homegrown. And then essentially, this audio is being fed into a speech-to-text system that converts it into text. That's being fed into an LLM that then decides, do I do a tool call? What is the output? Produces text that is then fed into a text-to-speech model, which starts to produce audio chunks that are then streamed back to the end user. So, that's a rough architecture. Let's maybe look at like each of the components one by one, the components that matter here. The first is speech to text. You know, very much like the ears of your agent here. The The performance metrics that matter here, the first is quality, word error rate. Now, you know, depending on use case, the numbers might look different, but state-of-the-art models are typically in the 6% word error rate on open benchmarks. What that really means is the transcript that is produced by your model, comparing it with the reference transcript, that 6% of words have an error in them. Now, you can imagine why this might matter, right? Because if you don't get the transcript right, you don't get somebody's name right, you don't get the name of, let's say, a drug right, essentially, there's no way to fix this. Your LLM will make the will carry forward the mistake. The TTS model will carry forward the mistake. So, you have to get it right for the important keywords. And then the second metric that matters often, which is latency driven, is time to complete a transcript. The way to understand this is when somebody completes an utterance, they stop speaking, how many milliseconds does it take for you to complete the transcript and have that be ready for the LLM? And so, as an example, for some of the models that we run on Together, we we get sort of consistently P90 of like 100 milliseconds, which is pretty fast. Aside from just raw performance, there's a few other capabilities that matter. Turn detection, very important, still somewhat unsolved problem, frankly. That could be a 20-minute talk in itself. But really, the best way to understand this is you've got people who are, you know, they're talking. Maybe they pause for a second. But do you actually know? Does that pause mean their turn has ended? Are they going to continue talking? Because really, the last thing that you want here is for you to for the agent to start, you know, sending audio back and talking at this person even though their turn hasn't ended. We don't enjoy this in human conversations, and we will certainly not enjoy this in AI conversations. Depending on who your customers are, language matters, and so being able to do this for a wide variety of language and getting it right there, it's important. And the final piece that I'll mention, this is somewhat new, is we're also starting to see architectures, model architectures that are streaming native. A little sidebar, we won't spend too much time on this, but there's an architectural evolution for speech-to-text models that is in progress, which is going from batch models to stream models. Whisper is the sort of canonical model came out a few years ago. Um, it was trained on 30-second audio clips. 30 seconds is way too much. You can't wait 30 seconds to start to do transcription. So, people have had to build all sorts of complicated logic around models like Whisper to do chunking and to pad it with silences and then it make multiple calls, stitch that together to produce the final transcript in in streaming mode. But recently, and this is a fairly new model from the Nvidia team, instead, you have the encoder of the model have two interesting characteristics. The first is it's trained with different amounts of look-ahead time. So, it only looks at perhaps 80 milliseconds or maybe up to a second of audio instead of 30 seconds. And it's also able to cache these activations so that as you kind of make small steps in audio frames, you're actually only doing the heavy computation once. So, again, just kind of stepping out, but, you know, it's an interesting direction that we're seeing to be able to handle streaming conversations for these voice agent use cases. Okay, so that's speech to text. Jumping into the next part of the pipeline, which is LLMs. You know, very much the brains of of your agent. The performance metric that matters here first and foremost is streaming latency. And so, you know, TTFT is the metric here. And a rough kind of, you know, metric is like it would it's usually pretty good if you can get to 300 milliseconds of TTFT because, you know, you want to start producing tokens, start feeding that into the TTS model as fast as possible. That number, 200 to 300 milliseconds, has implications for what models you can use. And so, a good size model typically ends up being in this 8 to 30 billion range. If you go any bigger, you'll burn through your latency budgets. If you go too small, that has implications for the intelligence of the model and, frankly, the tool calling that is needed, which are both pretty critical if you want to build a voice agent that does meaningful stuff in the world. Okay, text to speech. This is very much, you know, the voice of your agent. There's a few interesting things on on sort of performance and capabilities. Performance again, you know, the trend continues. What is the time to first audio, right? As you get a transcript, how long does it take to produce the first audio chunk that can start be start to be streamed back? And aside from TTFA, what is the real-time factor look like? Real-time factor is, and this is generally the case for most TTS models, but what it means is how much how much audio can you produce in certain number of seconds of processing time. So, if you can produce 10 seconds of audio in 5 seconds, your your RDF is 0.5. And so, you typically want that to be less than one so that you're not buffering. Um quality is one of the hard ones with TTS because, you know, there are some objective measures, but frankly nothing quite beats listening to audio samples for the models for the voices that you care about and getting a feel for is this the right uh experience that you want your end customers to have. Some of the other capabilities, you know, it's you know, naturalness across a different uh number of different voices, being able to pronounce things exactly right, whether it's, you know, customer names, whether it's product names, um being able to have some amount of control over emotions. And so, you might see TTS models that allow you to add these digital little tags which says, you know, this is happy or angry or sad. And it's the start, but these models are getting pretty good uh at emotional control. And of course, coverage over language continues to matter. Okay, so those are the main components, but just to kind of zoom out a little bit. You know, this is still All of these components are part of this larger architecture which is multiple models at being being orchestrated. And so, the there's a few things that we should always kind of keep in mind which is first, you know, there's a latency and cost budget across these models that we're thinking about. A rough rubric is the LLM is going to take up a majority of it followed by TTS followed by speech to text both from a latency and a cost perspective. Um and so, again, just kind of rough rules to kind of think about. And one piece that we didn't mention and we'll come back to this in a second is so far a lot of the the numbers that we're looking at is just engine latency. How much time does it take the model to produce an output? But actually, when you're calling models that might be sitting in different data centers, there's network latency as well and that starts to have an impact. But again, we'll come to that in um in one more slide. Auto scaling is also, you know, somewhat interesting and tricky to get right for for agent systems. Of course, you want to be doing auto scaling to scale up as demand goes up, scale down potentially, you know, night times or weekends you want to scale down seamlessly. Scaling up, you know, what we've typically seen as people are much more aggressive about scaling up because the last thing you want is, you know, request to be slowed down or backed up. So, you typically might auto scale earlier than you might do with somewhat more asynchronous systems. And scaling down is also tricky because you might actually have these stateful long-lived connections to your models. And so, you can't just arbitrarily kill a pod. You might want to wait for conversations to kind of fully finish. So, some interesting nuances with auto scaling. And finally, you know, it's like, you know, global deployments are important because you want your models uh you want your system to be as close to your end users to shave off latency as much as possible. And of course, you know, if you're building sort of models in Europe or in places where residency matters, you want to be making sure that you have the ability to deploy wherever you uh we absolutely need. I know I referenced co-location. Um but here's one way to kind of understand this problem. So, the chart on the left-hand side, this is very much, you know, a fairly optimized uh sort of system where you're doing a pretty good job with your speech to text and text to speech and LLM models where the engine latency is in exactly the right ballpark. You're doing 100 to 200 milliseconds of time to first token or audio. But you might actually end up having your models being sufficiently far away from your agent orchestrator that it's taking 75 milliseconds of network latency. 75 milliseconds is really not that much. Like, you know, even in I think like, you know, US West to Europe would certainly be 75 milliseconds, but you know, depending on networking it can be much higher as well. And so, an interesting kind of direction that we're seeing folks go is how can you co-locate all your models and potentially your agent orchestrator to either be in the same data center or be very very close to each other. How can you get them literally in the same building? Because that drop from 75 milliseconds to five basically gets you a 30% reduction in already a fairly optimized voice agent setup, right? So, some of these things um especially again with real-time systems uh it's just pretty important to have like fairly deep observability and, you know, every 10 milliseconds matters. Okay. So, hopefully that's an interesting picture on sort of this pipeline architecture. Um but that's not the only way people do it. One of the other kind of directions that is becoming interesting is a pure speech to speech model. And so, instead of having speech to text followed by LLM followed by text to speech where you're sort of coordinating and orchestrating across a number of different models, um it's just way simpler if you could have a pure speech to speech model that still is responsible for function calling, um still handles all of the complicated instructions, uh but just a single model doing it. And of course, you know, for folks who've played around with OpenAI's real-time API, they have a single model behind the scenes. Uh Nvidia recently launched a model called voice chat, again, very similar ideas. The reason why most of these models are not used in production too much is because they have they still have trouble with instruction following and tool calling. So, you know, the real-world experience often looks like you'll try them um and then you spend a lot of time just prompt engineering and hoping to kind of fix issues and, you know, eventually move to a pipeline architecture. But as these models get better, which I'm confident they will, uh it has some pretty incredible benefits because suddenly you don't lose anything about the nuances of speech when that speech is getting converted into text. So, the model will natively understand what was the tone, what was the emotion, uh was the user hesitant? That that stuff will still remain with the model to make the next kind of decision. And this type of model also allows for um sort of more full duplex homi- like communication which basically means that the model can start producing audio while it's still receiving audio. And this means that, you know, as your as a customer is talking to this model, you can back channel. You can say, you know, I see or aha like similar to what a human conversation would look like. And these models become much better at handling interruptions and bargins, which again, with a pipeline architecture you have to do a lot more complicated engineering work around. So, um hopefully this kind of points in the direction of like, you know, what might be coming in the future. But frankly, we have a lot of engineering work ahead for all of these kind of voice interfaces that still have to be built. Um if you want to learn more about what Together does, you know, uh here are a couple of links. We are hiring. We also have a booth G1 downstairs. Um happy to chat more and happy to answer questions. For voice to function calling use cases, what sort of um evals do you use? And then, what score do you need to get on those evals in order to have something that's good enough for production? Um that's like the classic answer is like it depends, you you know, but um I think in terms of evals like, you know, there's of course like component by component evals. So, if we're talking about sort of like the pipeline architecture um and assuming that speech to text is good, text to speech is good, and the only thing that we're caring about is sort of function calling or tool calling evals on the LLM, then it's very similar to how one might do evals for tool calling for LLMs broadly, which is, you know, was the um uh was the tool call correct? Was the output actually possible? There's a bunch of those. One would imagine that, you know, you'd want the uh the tool call structure to at least be very close to 100% uh um and then the correctness, again, it depends a little bit on what does the use case kind of eventually demand. One of the other things that we're seeing is especially to get around to make models better at tool calling and because we have to stay within that LLM budget of like, you know, the models have to be relatively small, we do see customers fine-tune smaller LLMs with their kind of use case specific data so that they can get tool calling quality to go up while remaining a model that is relatively small. Yeah. When you mentioned about co-location, so in that what I understand from that is you're using you're decreasing the latency in the network latency. Yeah. Um it's literally because the the machines are closer to each other. So, you \>\> that mean like if I'm using cloud providers or Uh good question. So, for example, um you know, let's say you're using let's say you're building a voice agent here in London, right? You have servers here, but you're using perhaps OpenAI's models for the LLM. And now, odds are that OpenAI's servers might be somewhere in the US. So, literally the data has the the network hop has to be from your server here all the way there and back. Uh instead, if you had if you're able to run, let's say, an open source model in the data center that you're running uh your voice agent, now it's basically going to be intra uh sort of uh like uh the data center rather than going um let's say over the Atlantic. So, that's one way to kind of think about like and just distance literally kind of like has that big of an impact aside from other kind of networking related concerns. Thank you. Yes. Uh one more question. Um So, we had uh when trying this voice pipeline system that we had to introduce some guard railing. So, like, let's say we have classifier model between uh which just checks that the model's not offering 8% discount uh which it's not authorized to do. Um How does it present to the pipeline and how you do that without compromising the latency and the the experience? Yeah, it's a great question. So, you know, the question is like what if we have other models like God real models or other classifiers in the mix? How does this fit into this architecture? You're absolutely right. Like this is the most kind of basic reference architecture that one might have. But in many production settings, there's actually multiple models that might be in the mix. The God real or the classifier, we definitely see people kind of like start to introduce that right before the main LLM as well because maybe you want to check is this something that um something that goes to an LLM that handles refunds versus something that handles order tracking perhaps. And so you might have a classifier there. God reals at the end of the LLM generation before you produce a response, that also makes sense. And so often this ends up growing as the as the sort of like scope of what you're hoping to achieve grows. And it puts real pressure on sort of like latency concerns and so forth. So, you know, no easy answers except that it becomes, you know, one more or two more components to think about. Have very clear sort of guidelines and SLAs on, you know, how much budget can you really kind of associated with them and then sort of independently scaling them as needed. But unfortunately, no no easy answers. Yes, especially because when you have things like like the the edge already Yeah. answer which was an answer which was already not what something it should be but the classifier the God reals were something Yeah. I should have lied. You can't take back things that are spoken. You you might have to do something Hey, sorry. I shouldn't have said that. I need to revoke that or something. So, this is really a problem I see there. Yeah, I I think that's spot on. Like catching all of those before the TTS model gets invoked is is certainly important. One of the other things patterns that at least like we've seen is sort of this thinker talker pattern where you might have a small LLM that is handling all of the conversation. And so as it gets sort of text from speech to text, it produces a response and the response might look like let me think about it or let me get back to you. And then it basically issues one big tool one tool call to a much bigger model that then has, you know, much you know, has better instructions, has all the tools associated, maybe more God reals. And then it produces a much cleaner response that you know, the the model is much more the the architecture is like, you know, you're much more comfortable with. And so that gets fed into the TTS model to produce a response. But you know, in some ways this is kind of the this is the kind of beauty of like all of these architectures is, you know, the components are, you know, you can we'll only add components. And so it kind of, you know, pushes more on sort of reliability, having like, you know, sort of detailed kind of observability on every single component. Yes. Um um I was I was just um curious about the upcoming voice voice conversations coming out. And I know that this is upcoming but at the end of the day all of the surrounding infrastructure that we have in the conversational while we have conversational systems now and all of yesterday pipeline approach, how do we like how do you do evals in the product and observability? Like is there a need to transcribe everything? Yeah, um very good question. So, how does sort of observability, logging, evals change? Um So, in some ways so some parts of it can still remain the same. So, sometimes what you might have is aside from pure speech to speech, you might actually have a a transcription model that is running so that you can at least seeing the transcription allowed at the same time as the audio is being generated. So, that gives you some amount of auditability in terms of what audio is coming in and what audio is being produced. But yes, you know, some evals are going to change. There is no there's no sort of real concept of sort of text to speech anymore. There's no concept of pure kind of text to text here anymore. And so the evals become much more full duplex kind of conversation evals which is like a much longer conversation and then, you know, evals and metrics uh that are sort of focused on that entire conversation. All right. But what I mean is like is the nature of those models are like able to output the characteristics of the the conversation in a way that you can evaluate is it that native to the pipeline or is that something on top? Typically a lot of the evals that would happen on top on top of the base kind of inference API. I think I think we're running out of time but Okay. Um thank you everybody for the for for being here.