Talk on building voice agents that feel human. It covers why model leaderboards don't predict pipeline performance, how errors compound across speech-to-text, the LLM, and text-to-speech, and the gap between measured and perceived latency. It also covers buying time during tool calls, a hotel-receptionist demo handling interruptions, and audio-based turn detection, where the speaker cites 4.5% of users cut off versus 9.9% for the best alternative. The speaker presents a latency window in which about 1.5 seconds end to end still works and about 600 milliseconds starts to feel human. The talk closes with an open-source benchmark for testing your own agent, a preview of a voice-tuned Gemma model, and a checklist. # Channel Presentation Observations: Single-Video Evidence **Source:** Video `01a15ca6-ad06-7e3c-811b-0d8b3f73d51e` **Title:** *Latency Is a Budget. Humanlike Is the Goal.*, Jesse Hall, LiveKit **Metadata:** None supplied beyond title and transcript. Speaker attribution comes from the title only. **Scope note:** These observations describe this one video. They do not support channel-wide conclusions. --- ## 1. Explicit Self-Description (Creator Claims) Claims the speaker makes about LiveKit and the talk itself: - **Positioning:** LiveKit is "the open-source real-time framework and infrastructure for the most human-like voice agents possible." - **Customer claim:** The speaker says "OpenAI actually chose LiveKit to power that" (referring to ChatGPT's advanced voice mode). This is unverified within the material. - **Neutrality claim:** > "we don't make STT, LLM, or TTS models. Every provider plugs into our stack. So we don't have a horse in this race." - **Openness claim:** The benchmark, "the methodology, the eval sets, the agent itself" is described as open source, and the speaker says "the eval harness is open source." - **Self-stated focus:** > "because voice is challenging, uh we're going to actually focus on voice during this talk." --- ## 2. Intended Audience (Demonstrated) The material addresses **builders and technical decision-makers working on real-time voice agents.** Evidence: - **Audience check:** The speaker asks, "who here has ever built a voice agent and shipped it to real users?" and whether attendees have "picked a model just because they saw it at the top of a leaderboard?" - **Practitioner framing:** Content centers on STT/LLM/TTS pipeline choice, latency measurement, turn detection, and tool-call handling. - **Actionable close:** A "checklist" for attendees to photograph, and a QR code linking to a benchmark that developers can "run against your own agents." --- ## 3. Claimed Scope The talk claims to address a bounded problem: - **Narrow modality:** Voice agents specifically. Video and "physical AI" are mentioned as LiveKit capabilities but explicitly set aside. - **Stated thesis:** > "latency it is a budget... The goal is being uh the voice AI being as human-like as possible." - **Explicit contrarian stance:** Speed-focused advice is labeled "the wrong answer." --- ## 4. Presentation Patterns Demonstrated in This Video ### Rhetorical structure - **Problem framing first:** Opens with contrast between background agents and real-time agents, and a relatable failure ("You talked over that AI"). - **Named concepts:** Introduces "the leaderboard trap" and "measured latency versus perceived latency" as frameworks. - **Live demonstration:** Plays a recorded hotel-booking call as evidence that "this is an entire system being good." ### Quantified claims (unverified within the material) - Interruption rate: **9.9%** for "the best alternative" versus **4.5%** for LiveKit's turn detector, "at the same latency budget." - Time to first audio: **~380 ms** for a Gemma/Gemini 31B model, described as "over two times faster than the next model." - Voice eval pass rate: **88%**. - Simulation scale: "hundreds of simulated callers." ### Product promotion - Includes a "sneak peek" of a model tuned for LiveKit inference, which the speaker says "is actually going to launch tomorrow." - Product and benchmark content is woven into the educational argument rather than separated from it. ### Tone - Conversational and direct, with audience participation (show of hands) and humor ("the most human thing possible"). - Confident in its own methodology: "You don't have to trust me, you can run your own data through this." --- ## 5. Transcript Quality Caveats - **Model name inconsistency:** The transcript says both "Gemini 4 31B" and "Gemma 4 31B." The correct name cannot be determined from this material. - **Formatting artifacts:** Escaped speaker markers (`\>\>`) and transcription artifacts appear throughout. - **Speaker attribution:** Only the title names the speaker. The hotel demo's caller is also named "Jesse Hall," but the material does not establish whether this refers to the speaker. - **Unverifiable claims:** Benchmark figures, the OpenAI customer claim, and the launch timing are all creator assertions without supporting evidence in the material. --- ## 6. Limitations - Single video, no curated metadata, no other channel content. - Cannot establish recurring format, publishing cadence, or consistent audience across the channel. - Cannot confirm whether the benchmark, eval harness, or agent repo exist as described. # Delivery Observations: "Latency Is a Budget. Humanlike Is the Goal." (Jesse Hall, LiveKit) > **Evidence base:** One transcript, supplied with the title. No curated metadata was provided, so every observation below rests on this single talk. Patterns are marked as stable only when they recur several times. Filler words and repetitions are reported as they appear in the transcript, which may not match the audio exactly. --- ## 1. Tone and Register ### Conversational and informal, with a technical core The speaker mixes casual phrasing with technical terms. > "They write code for us. They do research. Uh you give them a task, you walk away, you come back..." > "This is the infrastructure, the framework and the models all working together." ### Direct, assertive claims Several statements are framed as firm declarations rather than hedged suggestions. > "But that is the wrong answer." > "The leaderboards can't help you. The vendors definitely can't help you." --- ## 2. Vocabulary and Recurring Terms ### Core coined or repeated terms These terms recur across the talk and anchor the argument: - **"human-like"**: appears repeatedly, including: > "The goal is being uh the voice AI being as human-like as possible." > "This is the human-like behavior that we're looking for." - **"latency budget"**: introduced early and returned to near the close: > "latency it is a budget, and I'll I'll tell you exactly what that budget is." - **"leaderboard trap"**: named explicitly as a concept: > "and there's a name for this, the leaderboard trap." - **"measured latency" vs. "perceived latency"**: framed as the central distinction: > "This measured latency versus perceived latency. So, what your dashboard says versus what your user hears." ### Contrastive and reframing phrasing A recurring structure sets a common assumption against a corrected view: > "The question is never how fast can I go, but what is the most human-like agent that I can fit inside this window?" > "A voice agent isn't a chatbot with a speaker stacked onto it. It's human-like infrastructure." > "But bytes are not sound." > "Speed alone can get you none of that." --- ## 3. Verbal Habits and Recurring Phrases ### Tag questions seeking agreement "right?" and similar tags appear multiple times: > "They write code for us. They do research... right?" > "And they listen while they keep working. And that is that human-like behavior that we're looking for." > "It has to look up orders, check availability, book rooms, right?" Also present: "did it?" ("and it just didn't feel natural, did it?"). ### Filler words and repetition The transcript shows frequent fillers and stutter-like repetitions: > "Uh you give them a task..." > "Okay, so we had to speed up some of the parts there because of time, but the the the agent kept the conversation going." > "I'll I'll tell you exactly what that budget is." These appear throughout, so they are a stable feature of the transcript, though their prominence in the audio is uncertain. ### Sentence openers Sentences frequently open with "So," or "And so," used as connective transitions: > "So, let's talk about how we're going to fix that issue." > "And so, this is the distinction that this entire talk hangs on." > "So if we add all of this up..." --- ## 4. Teaching Habits ### Signposting The speaker repeatedly announces what comes next: > "So, let's talk about how we're going to fix that issue." > "So, let's go one level deeper for a minute." > "And here's the shape of that, and this is the only code that you're going to see." ### Audience participation Show-of-hands questions appear at three points: > "a show of hands quickly, who here has ever built a voice agent and shipped it to real users?" > "Another show of hands, who here has used ChatGPT's advanced voice mode..." > "show of hands here, who has picked a model just because they saw it at the top of a leaderboard?" ### Analogies and concrete scenarios The speaker grounds abstract points in everyday or specific examples: > "think about what a human receptionist would do here. They don't freeze while the booking system loads..." > "let's say that your caller wants to book an appointment on Tuesday the 3rd. Your speech-to-text hears Tuesday the 30th." ### Enumeration and structured summaries The talk uses explicit counts and checklists: > "So, here is what you have to choose from." > "So here's the checklist. You can take a photo of this. Set your latency budget first. Measure perceived latency." ### Demonstrating before explaining A recorded call is played to show the behavior before the speaker interprets it: > "Listen for yourselves." Followed by commentary: > "This is the human-like behavior that we're looking for." ### Inviting verification The speaker explicitly asks the audience to check claims independently: > "You don't have to trust me, you can run your own data through this and see what you get." --- ## 5. Communication Patterns ### Second-person address The speaker frequently addresses "you" and "your," both generically and to the audience: > "You probably did the most human thing possible. You talked over that AI." > "Your speech-to-text hears Tuesday the 30th." ### Meta-commentary on the talk itself The speaker comments on the presentation's own constraints and structure: > "So we had to speed up some of the parts there because of time." > "We'll talk about that more in a bit." ### Closing call to action The talk ends with an imperative and a forward-looking statement: > "So go build that. Thank you." Preceded by a prediction framed as expectation: > "at some point in the near future, you are going to call a voice agent, and you're going to talk over it, and it's not going to fall apart." --- ## 6. Uncertainty and Limits - **Single source.** With one talk, recurring phrases can be identified, but it is not possible to confirm whether they are habitual across the speaker's other presentations. - **Transcript fidelity.** Fillers, repetitions, and at least one apparent transcription error (for example, "rhyme" where "rhythm" is likely intended, and "Gemini 4 31B" where "Gemma 4 31B" appears later) suggest the text is machine-generated. Delivery observations about fillers should be verified against audio. - **Audience and recording conditions** are not visible in the transcript, so claims about audience response are excluded. # Observation Record: "Latency Is a Budget. Humanlike Is the Goal." **Source:** Video `01a15ca6-ad06-7e3c-811b-0d8b3f73d51e` **Speaker:** Jesse Hall, LiveKit (speaker identified from the video title; no other speakers appear in the transcript) **Nature of content:** Conference-style presentation. Includes vendor-originated performance claims and a product announcement, which are noted where they appear. --- ## 1. Core Thesis - **Central claim:** The goal for voice agents is to be *human-like*, and latency is a *budget* to be managed toward that goal, not the goal itself. - **Explicit rejection:** Hall states that the common industry answer, "speed," is "the wrong answer." - He says speed alone "can get you none of that" (listening while talking, knowing when a thought is finished, adapting when a caller changes their mind). - He attributes these capabilities to **model selection and pipeline configuration**. - **Framing statement:** > "A voice agent isn't a chatbot with a speaker stacked onto it. It's human-like infrastructure." --- ## 2. Voice Agent Architecture (As Described) - Voice agents are described as a **cascade of three models** working as one system: - **Speech-to-text (STT)** - **LLM** - **Text-to-speech (TTS)** - Hall acknowledges **real-time (speech-to-speech) models** exist but says they are "not right for every use case" and add complexity to model selection. --- ## 3. Model Selection ### 3.1 Claimed Problem - **"No one tells you how to pick the right models."** - Leaderboards "can't help." - Vendors "definitely can't help." ### 3.2 The Leaderboard Trap - **Argument:** Leaderboards score *components*, but the product is a *pipeline*. - **Illustrative example (hypothetical, presented by speaker):** A caller wants Tuesday the 3rd; STT hears Tuesday the 30th. Downstream components have no way to recover, so "every error flows downhill." - **Environmental mismatch:** Leaderboard scores come from "clean audio in a quiet room," whereas callers use speakerphones, traffic, and so on. - **Reframing:** - Common question: *"What is the best STT, LLM, or TTS model?"* - Hall's proposed question: *"Which stack should I ship?"* - He states these are "two completely different questions," and that only the first has a leaderboard "until today," a reference to the benchmark he presents. --- ## 4. Measurement: Measured vs. Perceived Latency ### 4.1 Core Distinction - **Measured latency:** What dashboards report. - **Perceived latency:** What the user hears. - Hall says "this is the distinction that this entire talk hangs on." ### 4.2 TTS - Providers commonly report **time to first byte (TTFB)**. Hall argues: *"bytes are not sound."* - **Claimed finding (from the speaker's own measurements of "popular providers"):** - Hundreds of milliseconds of leading silence in some providers' audio. - In the worst case, **up to 3/4 of a second** of silence before first audio. - **Proposed metric:** *Time to first audio* is what the caller actually experiences. ### 4.3 LLM - **Time to first token** is a text metric, but TTS needs a **complete first sentence**, not a token. - A model that streams tokens fastest "is not necessarily the model that finishes a sentence first." --- ## 5. Tool Calls and Asynchronous Behavior - **Premise:** Real agents must act (look up orders, check availability, book rooms), and tools can take **2 seconds to 40+ seconds**. - **Claim:** "No model choice is going to be able to hide a 40 or more second latency." - **Human analogy:** A receptionist says "Give me a sec," buys time, and keeps listening. - **Proposed mechanism:** - A normal tool with one difference: the *first update* returns the microphone to the agent while work continues in the background. - Hall summarizes: "The conversation and the tool stop sharing a thread." - **Demonstration:** A recorded hotel-booking call was played. Hall states it was "speed[ed] up" in parts for time. Observed behaviors (as described by the speaker): - No awkward pauses. - The caller interrupted, and the agent continued. - The caller changed the dates mid-booking, and the agent adapted. - **Interpretive claim:** "This is not one model being good, this is an entire system being good." --- ## 6. Turn-Taking - **Premise:** Humans negotiate turn-taking in "a couple of a hundred milliseconds, mostly from tone." - **Critique:** Pipelines that wait for the full transcript are "already late." - **Proposed approach:** A turn detector based on **audio itself** (rhythm, pitch, intonation), rather than transcript text. - **Claimed results (vendor-originated, at the same latency budget):** | Detector | Caller cut-off rate | |---|---| | Best alternative (unnamed) | 9.9% | | LiveKit's | 4.5% | - **Verifiability claim:** Hall states "the eval harness is open source" and invites others to run their own data. --- ## 7. The Latency Window - **Mechanism described:** - End-of-turn detection fires from audio *before* the transcript finalizes. - The LLM begins on a **provisional transcript**; if final words match, roughly **half a second is saved**. - TTS begins on the first complete sentence. - **Definition:** The "first audible bite" marks when the human stops waiting. - **Stated thresholds (qualified):** - **Within ~1.5 seconds end to end:** "the conversation still holds, but you're right on the edge." - **Approaching ~600 ms:** "starts to feel really human." - **Key reframing:** "The question is never how fast can I go, but what is the most human-like agent that I can fit inside this window?" --- ## 8. Benchmark Methodology ### 8.1 Design - **Reference agent:** A hotel receptionist agent with real bookings and a real back end. Described as open source. - **Simulation:** Hundreds of simulated callers who interrupt, mumble, and change their minds. ### 8.2 Scoring - **Not "on vibes."** Outcomes are scored against "human equivalent behaviors": - Correct reservation in the database? - Caller cut off? - Names and dates correct? - Invented tool calls? - Leaked personal data? - **Anti-gaming:** Completion alone is considered "gameable." Penalties apply for: - Unnecessary questions. - Ten steps where two would suffice. ### 8.3 Comparability - The stack is swapped while holding scenarios and inputs constant: "Every model sees the exact identical input byte for byte." - Hall notes benchmarks are **agent-specific**: "There are as many benchmarks as there are agents." He encourages users to run their own agents. --- ## 9. Vendor Claims and Product Announcement > **Note:** The following are the speaker's claims about LiveKit's own model work. No independent verification appears in the transcript. - **Neutrality claim:** "LiveKit... [doesn't] make STT, LLM, or TTS models. Every provider plugs into our stack. So we don't have a horse in this race." - **Adoption claim:** OpenAI "chose LiveKit to power" ChatGPT advanced voice mode. - **Announced model:** A tuned variant of a Gemma-family model, described as: - Tuned for "production voice agents with long prompts, many tools, and tight latency budgets." - Passes voice evals at **88%**. - Starts speaking in about **380 ms**. - Described as "over two times faster than the next model." - Called "the actual the fastest high-quality voice LLM available today." - **Timing:** Launch on LiveKit inference is stated as "tomorrow" (relative to the talk). --- ## 10. Recommendations (Speaker's Checklist) 1. **Set your latency budget first.** 2. **Measure perceived latency**, not just dashboard metrics. 3. **Spend remaining budget on model quality**, not on speed that doesn't matter. 4. **Anything over one second** should run asynchronously so the conversation never stops. 5. **Test the conversation, not the components.** 6. **Keep testing in production.** --- ## 11. Tensions, Qualifications, and Open Points - **Speed vs. "speed alone":** Hall says speed alone is insufficient but also treats latency as a binding constraint and advocates reducing it toward ~600 ms. These are presented as complementary (budget vs. goal), not contradictory. - **Quantitative thresholds are not universal:** The 1.5 s and 600 ms figures are presented as guidance on a "window," without stated derivation in the transcript. - **Self-reported and unverified:** - Silence measurements of unnamed "popular providers." - Turn-detector comparison (the alternative is unnamed). - The 88% eval score and "two times faster" claim. - The "fastest high-quality" superlative. - The hotel demo was edited for time, which the speaker discloses. - **Hypothetical example:** The "Tuesday the 30th" scenario is illustrative, not a documented incident. - **Commercial interest:** The speaker represents LiveKit, which sells infrastructure and inference, and the benchmark and model are LiveKit's. Hall states the methodology and eval sets are open source; the transcript does not confirm independent replication. - **Speaker's own caveat:** Real-time models are "not right for every use case." --- ## 12. Transcript Notes - The transcript contains apparent transcription errors that were not corrected here: - "Gemini 4 31B" and "Gemma 4 31B" both appear; the speaker's intended model name is ambiguous in this transcript. - "the rhyme, the pitch" likely refers to *rhythm*. - "Listen for yourselves" appears as "It Listen for yourselves." - Timestamps are not present in the source and are not supplied here. # Attribution Observation: Latency Is a Budget. Humanlike Is the Goal. **Video ID:** `01a15ca6-ad06-7e3c-811b-0d8b3f73d51e` **Source basis:** Transcript and title only. No curated metadata, speaker list, comment data, or timestamps were supplied. --- ## 1. Speakers and Roles ### Primary speaker - **Jesse Hall**, identified as affiliated with **LiveKit** by the title only. - **Role:** Conference presenter giving a talk on voice agents. - **Speaking style:** Single continuous speaker, addressing an audience (show-of-hands prompts are used). - **Note:** The transcript does not state the speaker's title. Affiliation rests on the title. ### Embedded audio demonstration - **Unlabeled voices** in a recorded hotel booking call, introduced as "Listen for yourselves." - **Agent:** Automated hotel receptionist voice ("Live Oak Hotel"). Described by the speaker as an example agent built on the presenter's organization's stack. - **Caller:** Simulated or recorded caller. Gives the name "Jesse Hall" within the scripted dialogue. - **Caution:** The caller's name in the demo matches the presenter's name. The transcript does not say whether the presenter is the caller. Do not treat them as the same person without further evidence. ### Other voices - **Audience members** respond to show-of-hands prompts. Their responses are not transcribed individually. - **Unidentified references:** "anybody in this industry," car dealership and airline agents, and "a pharmacy refill." These are anecdotal and not attributed to named individuals. --- ## 2. Referenced Sources and Entities ### Named organizations and products - **LiveKit:** Presenter's organization. Described as an open-source real-time framework and infrastructure. - **OpenAI:** Named in the claim that it "chose LiveKit to power" ChatGPT advanced voice mode. - **ChatGPT advanced voice mode:** Referenced as a product users may have used. - **Gemma / Gemini 4 31B:** Model named as the subject of a sneak peek. - **Discrepancy:** The transcript says "Gemini 4 31B" once and "Gemma 4 31B" elsewhere. Record both as written; the correct name is unresolved. - **LiveKit Inference:** Hosting platform where the model is said to launch. ### Open-source artifacts referenced - **Hotel demo agent repository:** Described as open source; "you can look at the repo." - **Turn detector evaluation harness:** Described as open source. - **Benchmark methodology, evaluation sets, and agent:** Described as open source. - **Benchmark access:** A QR code is displayed. The transcript does not state the URL. ### Unnamed sources - **Leaderboards:** Referenced repeatedly as ranking models. No specific leaderboard is named. - **Vendors and TTS providers:** Referenced without names. The speaker cites unnamed "popular providers" for silence measurements. - **Measurements of popular providers:** The speaker says "we've measured" silence at the start of TTS audio. No dataset or methodology is cited in the transcript. --- ## 3. Third-Party Claims These claims are attributed to other parties by the speaker. The transcript does not provide supporting evidence within the video. - **OpenAI selected LiveKit** to power ChatGPT advanced voice mode. - **Leaderboard trap:** The speaker asserts leaderboards score components, not full pipelines. - **Vendor dashboards:** The speaker says TTS providers report time to first byte, and that some ship hundreds of milliseconds, up to about 0.75 seconds, of silence at the start of audio. --- ## 4. Self-Reported Claims by the Presenter's Organization These are performance claims made by the presenter's organization. They are **not independently verified** within the transcript. - **Turn detection comparison:** At the same latency budget, the best alternative cuts users off **9.9%** of the time; the presenter's detector cuts them off **4.5%**. - **Gemma/Gemini 4 31B on LiveKit Inference:** - Passes voice evals at **88%**. - Starts speaking in about **380 ms**. - Described as "over two times faster than the next model." - Described as "the fastest high-quality voice LLM available today." - **Benchmark design:** Hundreds of simulated callers; outcomes scored against "human equivalent behaviors." - **Demo editing disclosure:** The speaker says the demo was **sped up in parts** "because of time." This is an explicit provenance fact about the recording. --- ## 5. Promotional Segments (Flagged) The following segments promote the presenter's organization, products, or tools. > **Promotion 1: Product description.** LiveKit is described as "the open-source real-time framework and infrastructure for the most human-like voice agents possible." > **Promotion 2: Customer claim.** The OpenAI association is presented as evidence of platform adoption. > **Promotion 3: Product launch.** The Gemma/Gemini 4 31B model is previewed as launching "tomorrow on LiveKit inference." > **Promotion 4: Benchmark call to action.** Attendees are directed to scan a QR code and "run this against your own agents." > **Promotion 5: Closing recommendation.** "Set your latency budget first," followed by the speaker's checklist, centers the presenter's framework and benchmark. **Disclosure:** The speaker states that LiveKit "doesn't make STT, LLM, or TTS models" and has "no horse in this race." This is a self-description of neutrality. The transcript does not verify it. --- ## 6. Availability and Limits - **Comments:** No comment data was supplied. - **Timestamps:** None supplied. Segment boundaries cannot be cited by time. - **Curated metadata:** None supplied beyond title and transcript. Speaker identity and affiliation rest on the title. - **Transcript artifacts:** Encoded characters (for example `\>\>` for speaker markers) and a `[music]` tag at the end are present. - **Audio demo:** Speaker identities within the recording are not labeled, and the speaker notes that the segment was accelerated. - **Unverifiable items:** Performance figures, adoption claims, and comparisons to unnamed providers cannot be checked against this transcript alone. All right, let's get started. So, most of the agents that we build today, they do their work in the background, right? They write code for us. They do research. Uh you give them a task, you walk away, you come back, and so nobody really cares if uh a background agent takes an extra 10 seconds or even 10 minutes, right? We're used to that. But, in at some point in this past year, you've probably talked to another kind of agent. A real-time agent. Uh maybe uh it was a pharmacy refill, maybe at an airline. Um I talked to one the other day when I called a car dealership. And and so something happened on that call. You probably did the most human thing possible. You talked over that AI. And that voice agent fell apart, and it just didn't feel natural, did it? And so um if you ask uh anybody in this industry, you're probably going to get a very similar answer of how to fix that. It's going to be speed. Uh faster models, lower latency, shave off another 100 milliseconds. Um but that is the wrong answer. Uh and so latency it is a budget, and I'll I'll tell you exactly what that budget is. But, the goal is something else entirely. The goal is being uh the voice AI being as human-like as possible. Uh an agent that listens while it talks. Uh one that knows the difference between you finishing your thought and you still thinking. Uh one that rolls with it when you change your mind. And so speed alone can get you none of that. What gets you there is which models you pick and how you configure your pipeline. And so that's the first problem that we're going to encounter. No one tells you how to pick the right models. The leaderboards can't help you. The vendors definitely can't help you. So, let's talk about how we're going to fix that issue. So, a show of hands quickly, who here has ever built a voice agent and shipped it to real users? Okay. All right. Some of you have experience and you know the challenges that you'll face when deploying a real-time agent. Another show of hands, who here has used ChatGPT's advanced voice mode to talk with ChatGPT? Yeah, many of us have have used that, right? So, OpenAI actually chose LiveKit to power that. LiveKit is the open-source real-time framework and infrastructure for the most human-like voice agents possible. Uh agents that can hear, see, and speak with the person on the other end. Uh and so, not just voice, but also video, uh data, and even physical AI. But, because voice is is challenging, uh we're going to actually focus on voice during this talk. And so, um what we're going to uh look at is is voice agents. They basically have um three models uh that work together as one system. So, your speech-to-text, your LLM, and then your speech uh text-to-speech. So, three slots. Now, I do have to acknowledge that there are real-time models as well. And and real-time models, uh they are a newer concept. They're getting better and better, uh but they're not right for every use case. And just the fact that there are real-time models even adds more complexity to our problem of trying to figure out which model is best for our voice pipeline. So, here is what you have to choose from. Hundreds of models, dozens of providers, uh and show of hands here, who has picked a model just because they saw it at the top of a leaderboard? Yeah, most of us. Um but here's why that doesn't work for real-time agents. Leaderboards score components, but you are shipping a cascade, a pipeline. So, let's say that your caller wants to book an appointment on Tuesday the 3rd. Your speech-to-text hears Tuesday the 30th. And so, from that moment on, it doesn't matter how smart your language model is, it's now confidently booking the wrong day, and every error flows downhill from there, and the layers below it have no idea. And so, those leaderboard scores, they came from clean audio in a quiet room. Uh but your callers, they're calling on a speakerphone, in traffic, with kids in the backseat. So, those leaderboards never met your users. Uh and there's a name for this, the leaderboard trap. So, this the score measures the model, but again, you're shipping a conversation. But everyone building voice agents, they're asking, you know, what is the best speech-to-text model, LLM, TTS? But the real question is, which stack should I ship? So, the these are two completely different questions, and the only only one of them has a leaderboard, until today. Uh we're going to talk about that more in a bit. So, let's say you fall into this trap, you pick your models based on the leaderboard, and now you start measuring things. This is the next problem. You're probably measuring the wrong thing. You see, your TTS provider uh reports time to first byte, and that looks great on the dashboard, uh but bytes are not sound. And so, we've measured popular providers that are actually shipping hundreds of milliseconds of silence at the beginning of their audio. And in the worst case, up to 3/4 of a second of nothing but silence before that first audio comes in. So, time to first audio is actually what your caller hears. And so, this is the distinction that this entire talk hangs on. This measured latency versus perceived latency. So, what your dashboard says versus what your user hears. The dashboard says it's really fast, but your ears say it's slow. Same story for language models. Time to first token is a text metric. So, your TTS can't start speaking though on a token, right? So, it needs an entire sentence. So, that that number matters for for voice because it needs the entire first sentence, but the model ranks differently on that. So, a model that streams back back a model that streams tokens the fastest is not necessarily the model that finishes a sentence first. Now, if if you let's say you do measure all of this perfectly, there's another part of the conversation that the benchmarking that gets completely ignored. And that is that a real agent doesn't just chat. It needs to do things, right? It has to look up orders, check availability, book rooms, right? And so, a tool is going to take time. 2 seconds, 10, 40, or more seconds. And no model choice is going to be able to hide a 40 or more second latency. So, think about what a human receptionist would do here. They they don't freeze while the the booking system loads, right? They say, "Give me a sec. I'm looking up your your account, right?" They they buy time. And they listen while they keep working. And that is that human-like behavior that we're looking for. So, why can't an agent just keep talking and do things asynchronously? Well, it can. And here's the shape of that, and this is the only code that you're going to see. Uh but this is a normal tool. Uh there's just one thing different, and that is this line right here. Uh the first update hands the mic back to the agent while it keeps uh working in the background. And that's the whole trick. The conversation and the tool stop sharing a thread. Uh and so, don't take my word for it. It Listen for yourselves. \>\> Thanks for calling the Live Oak Hotel. How can I help you? \>\> Can I book a room? \>\> Of course. What dates are you looking at? \>\> October the 10th through 12th. \>\> October 10th to 12th? And for how many guests? \>\> Two. \>\> I have queen with two beds, king, double, queen, or suite. Which Would you prefer a city or ocean view? \>\> Ocean. \>\> Who should I put the booking under? \>\> Jesse Hall. \>\> You're all set for October 10th to 12th in a king room with an ocean view for $582.40 on the card ending in 1111. Shall I go ahead? \>\> Actually, you know what? I need to change the date to December the 20th through the 24th. \>\> December 20th to 24th? That brings the total to $1,164.80 for December 20th to 24th. Shall I go ahead? \>\> Yes. \>\> You're all set. \>\> Okay, so we had to speed up some of the parts there because of time, but the the the agent kept the conversation going. There were no awkward pauses. The caller even interrupted it and it kept going. And the caller also changed his mind in the middle of things and the agent just rolled with it, right? This is the human-like behavior that we're looking for. So this is again not one model being good, this is an entire system being good. This is the infrastructure, the framework and the models all working together. And that's exactly what no component or model benchmark can see. Now there's one more axis to this that the leaderboards miss entirely and this is knowing when to speak and when not to speak. So humans negotiate turn taking in a matter of just a a couple of a hundred milliseconds, mostly from tone. And a pipeline that it waits for the entire transcript to come back, it's already late, right? So we built a turn detector that is based on audio itself. So it listens to the rhyme, the pitch, the intonation of the speech of the audio and then we published the numbers on this. So at the same latency budget, the best alternative cuts users off 9.9% of the time. Ours is 4.5% and the eval harness is open source. You don't have to trust me, you can run your own data through this and see what you get. So let's go one level deeper for a minute. The gap between your caller's last syllable and your agent's first audible bite. So end of turn detection fires from audio before the transcript lands, the language model is already running on a provisional transcript and if the final words match then you just saved a half a second. And then the TTS starts on the first complete sentence, not the last token. And so the the the first audible bite is when the human stops waiting. And so every model that you choose, it shows up here in this window, and this window has a fixed size. If you can stay within a second and a half end to end, the conversation still holds, but you're you're right on the edge. If you can get closer to 600 milliseconds, then it starts to feel really human. And so the question is never how fast can I go, but what is the most human-like agent that I can fit inside this window? So back to latency as a budget, but human-like is the goal. So if we add all of this up, the question that we asked at the beginning, what models should I pick? That was never actually answerable. And so we built the thing that we wish existed. Here's how we built the benchmark. We take a complete working agent. The first one is that hotel receptionist that you just heard. And it it takes real bookings. It has a real back end. And it's also open source. You can look at the repo. And we run hundreds of simulated callers against it. The the simulator interrupts. It mumbles. It changes its mind, just like a real caller would. And then we don't score it on vibes, but we score outcomes defined the way that we would define them for a human. And so did the right reservation land in the database? Did a caller get cut off? Are the names and dates correct? Are there were there any invented tool calls or any leaked personal data, right? So we measure all of those things against human equivalent behaviors. That's the bar. And then completion alone isn't enough, because completion can actually be gameable, right? So a model can kind of guess its way to complete, right? And so we penalize what a completion a completion score can't see. If like if it asks questions that didn't need to be asked or if it takes 10 steps to do something that should have only taken two steps. And then we swap the stack out. The same agent, the same scenarios, but different models. Every model sees the exact identical input byte for byte. But this creates benchmarks against our agent example. There are as many benchmarks as there are agents. So the benchmark is actually yours. You can run this exact thing run your agent through this exact benchmark and get your scores. And so this is how you catch surprises. The whole thing is open source, the methodology, the eval sets, the agent itself. You can point the same thing at your agent and let us know how it performs for you. Now here's an important fact about LiveKit is that we don't make STT, LLM, or TTS models. Every provider plugs into our stack. So we don't have a horse in this race. We just watch all of them run a lot. And so we can confidently pick the winners. And so what you're looking at right here is a sneak peek of the Gemini 4 31B model on LiveKit inference. But this isn't just an off-the-shelf Gemma 4 31B. It's tuned specifically for production voice agents with long prompts, many tools, and tight latency budgets. It passes voice evals at 88% and it starts speaking in about 380 milliseconds. So that's over two times faster than the next model. Making this the actual the fastest high-quality voice LLM available today. And so this is actually going to going to launch tomorrow on LiveKit inference. So again, our goal is to have the most human-like agents possible while sticking to that latency budget. So if you remember one thing from this talk, make it this one. A voice agent isn't a chatbot with a speaker stacked onto it. It's human-like infrastructure. A real-time system that senses and responds the way that a human, a person, would. The moment that you treat it that way, every decision is going to get clearer. The budget, the metrics, the models, and the bar that you test against. So here's the checklist. You can take a photo of this. Set your latency budget first. Measure perceived latency. Spend whatever budget is left on model quality, not on speed that doesn't matter. And then anything over a second it should go asynchronously so that the conversation never stops. And then just test the conversation, not the components. And then keep testing in production. So here's the benchmark. Uh scan the QR code. You can check out the benchmark. You can run this against your own agents. And at uh remember at some point in the near future, you are going to call a voice agent, and you're going to talk over it, and it's not going to fall apart. It's going to act like a human would in that scenario. So go build that. Thank you. \>\> \[music\]