Talk by Bohan Li (EliseAI) on building realtime voice agents that use slow, highly capable frontier models while keeping latency unnoticeable to callers. Li frames the cascaded voice stack using a self-driving analogy: transcription as perception, the language model as planning, and speech synthesis as control, with each layer optimized independently. Techniques covered include running a fast streaming transcriber alongside a slower, higher-accuracy one (with late corrections discarded when newer audio has arrived), eager generation with background tool calls that inject results into the model's context, and a prefix cache that lets synthesis start playing already-generated audio while the sentence is still being written, with the seam between cached and newly synthesized audio hidden. The talk closes with a real clinic booking call demonstrating the system. # Channel Identity Observations — Realtime Voice Agents with Frontier Intelligence ## Explicit Self-Description - **Company affiliation stated by speaker**: The presenter identifies himself as "Bo" presenting "real-time voice agents with Frontier Intelligence" and describes work done "at Xnor.ai" on a voice agent harness. Note: the title attributes the speaker to **EliseAI**, and the closing section describes EliseAI in first person ("our headquarters are in New York"). - **Company self-characterization**: Near the end, the speaker describes EliseAI as "maybe like a different style of company… when they think about AI startups in San Francisco. Where we're actually very focused on just like helping people… in life's most critical areas. We work on housing, health care." - **Recruiting intent stated directly**: "there's a link here to kind of join our team and… we're going to be posting a lot on Twitter, so you can follow us at EliseAI." ## Claimed Scope - **Technical scope**: Architecture of a **cascaded (not end-to-end) real-time voice agent**, framed as a three-layer pipeline analogous to self-driving cars: perception (transcription), planning (LLM), and control (text-to-speech). - **Stated goal**: "get real-time voice with the Frontier level of intelligence that we need" and "a lot of effort… in the harness to really kind of get a natural conversation, which is what we're after." - **Named components/techniques** (all introduced in-transcript): - Streaming speculative transcriber layering Flux over "scribe V2" batch transcription - Background agents performing tool calls on behalf of a "slow but intelligent" LLM - Prefix cache in the TTS layer, integrated with Cartesia over websockets ## Intended Audience - **Engineering/technical audience**: Deep implementation detail (streaming cancellation, cache hits/misses, audio suppression, phonetic matching) delivered as a talk with slides and a recorded demo call. - **Prospective hires and followers in the Bay Area**: Explicit hiring pitch and Twitter follow prompt at the close. - **AI-conference/meetup context**: Presented format ("I'm going to be here presenting…", closing applause, "\[applause\]") suggests a live technical event. ## Observable Presentation Patterns - **Demo-driven proof style**: Includes a full recorded phone call demonstrating scheduling at "Bo OBGYN Elise," handling pregnancy-related scheduling, insurance upload links, and calendar negotiation — demonstrating a healthcare use case rather than merely claiming capability. - **Domain framing via product example**: The demo uses a healthcare reception scenario, consistent with the stated company focus on "housing, health care." - **Speaker background used as framing device**: Cites prior self-driving car work ("which I was working in before") to structure the voice-agent architecture analogy — a stated credential, not independently verified. ## Claims vs. Demonstrated | Claim | Evidence type | |---|---| | Real-time, intelligent voice agent architecture | Explained in detail with step-by-step walkthroughs | | Fast, natural-sounding conversation | Demonstrated via recorded demo call transcript | | Company focus on housing/healthcare | Stated by speaker; healthcare shown in demo; housing not shown | | "We're doing really well" | Unsubstantiated claim, no supporting evidence in material | **Note on ambiguity**: The transcript says "we at Xnor.ai architected our voice agent" while the title attributes the speaker to EliseAI and the closing discusses EliseAI — this discrepancy is present in the source material itself and is reported as-is without inference. # Delivery Observation: "Realtime Voice Agents with Frontier Intelligence" — Bohan Li ## Overall Delivery Style The talk is delivered in a **conversational, engineer-to-engineer register** — informal, walkthrough-driven, and heavy on live demos of the system architecture rather than polished slides-read-aloud narration. ## Tone - **Casual and unrehearsed-feeling**: frequent filler and self-narration of live actions. - "So, um, I'm going to kind of play the streaming um the stream the streaming uh agent output now." - "Um yeah, so here I'm going to be like going to diving into each one of these elements..." - **Mild enthusiasm around demos**: "you know, a lot of things are happening in the background and and you know, this is what really makes like voice agents interesting." ## Vocabulary & Framing Habits - **Analogy-driven explanation**: opens with an extended self-driving car analogy — perception / planning / controls mapped onto transcription / LLM / TTS. - "cascaded voice agents makes a lot of sense when you view it in lens of kind of breaking it down into perception..." - **Consistent three-layer terminology**: "transcriber layer," "language model layer," "text-to-speech" — repeated as organizing structure throughout. - **Product/vendor names used freely**: "Flux," "Scribe V2," "Cartesia" — assumes a technically fluent audience. ## Recurring Phrases (most frequent) - **"kind of"** — extremely dense filler, appears dozens of times ("we've made a few kind of interesting tricks," "it's kind of hiding the latency"). - **"you know"** — pervasive connective ("the bounding boxes, you know, the camera, the lidar"). - **"So,"** as a paragraph-opener: "So, the first one is going to be uh the transcriber layer." - **"effectively"** — recurring summarizer: "effectively turning these like signals from the real world into elements of data." ## Teaching Habits - **Walkthrough-by-scenario**: repeatedly narrates a timed example ("So, we're going to walk through a scenario... we'll see how that plays out um timing-wise") rather than describing abstractions alone. - **Step-by-step incremental narration**: "So, first we're going to get, you know, the short detection... Then a couple more detections... And so, in the end we kind of release this text." - **Demo-as-proof**: "I mean the best way is just kind of play a call and see see this in action" — follows with a full recorded phone call. - **Preemptive acknowledgment of detail overload**: "I guess at this point kind of this is a lot of technical details." - **User-experience framing for technical tricks**: repeatedly lands on how it appears to the end user — "it looks seamless to the user," "the user doesn't really know what's going on. They just looks like really fast response times to them." ## Structure & Closing Pattern - Signposted transitions between layers ("And moving on um to the language model layer," "Okay, and then the next layer here is going to be text-to-speech"). - **Ends with a company pitch**: "I want to just talk a little bit about Elise... there's a link here to um you kind of join our team." ## Uncertainty Notes - Filler density may be inflated by **auto-transcription of disfluencies**; exact frequency of "um"/"uh" should be treated as approximate. - No visual delivery cues (slides, pacing, audience reaction beyond final applause) are available from the transcript alone; observations are limited to linguistic content. # Observation Document: "Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI" ## Speaker Identification - **Bo (Bohan Li)** — sole presenter; introduces himself as "Bo," previously worked in self-driving cars, presents on behalf of "Xnor.ai" (as spoken in transcript). > Note: The transcript's speaker self-identifies the work as done "at Xnor.ai"; the video title attributes him to EliseAI. Both attributions are preserved as expressed; the discrepancy is not resolved here. --- ## Position: Cascaded Voice Agents Are the Right Architecture - **Claim (Bo):** Cascaded voice agents make sense when broken into three layers, analogous to self-driving cars: - **Perception** — self-driving: bounding boxes, camera, lidar; voice: **transcription**, turning real-world signals into data the "brain" (LLM) can process. - **Planning** — the language model consumes perception outputs and produces outputs for the real world. - **Controls** — self-driving: converting planner trajectories into driving controls; voice: turning text into audio expressing the agent's thoughts. - **Claim:** Each layer incorporates tricks to **improve speed without sacrificing intelligence**. --- ## Layer 1: Streaming Speculative Transcriber - **Claim:** A fast streaming transcriber (**"Flux"**) is layered below a slower, more accurate batch transcriber (**"scribe V2"**) that uses more context. - **Reasoning via walkthrough scenario** (agent asks for name and date of birth): - Short detections arrive from the streaming layer. - The corrective layer is **canceled** when new streaming text arrives, because "more context, more audio is going to beat the old accurate one." - The corrective layer fires on a correction because it understands the **context of the question** (name, date of birth). - Punctuation-only detections are ignored. - Final corrected text is released to the agent. --- ## Layer 2: Language Model / Background Tool Calling - **Claim:** Since they use "slow but intelligent LLMs," the goal is to **reduce round trips**; tool calling is the main driver of extra inference. - **Mechanism:** **Background agents** perform tool calls and push tool results back into the main agent's context "so that it thinks it made the tool call, but it really didn't." - **Reasoning via walkthrough:** - Each detection triggers **eager early generation** of the agent, but output is held until the user is confirmed finished speaking. - Early generations are speculative; the agent may ask the user to spell out a name if it suspects a transcription error. - When the corrected final detection (from scribe V2) arrives, the eager generation **without tool calls is canceled** because the background agent found the name and date of birth; the agent retriggers with full context. - Tool calls include "some intelligence": correcting name mis-transcriptions and **phonetic matching**. --- ## Layer 3: Text-to-Speech with Prefix Cache - **Claim:** The agent streams text; the goal is to produce audio as fast as possible, ideally **playing audio before generation completes**, hiding generation latency. - **Prefix cache concept (new, as introduced):** - Looks at the agent stream for word sequences whose audio was already generated (prior generations or earlier in the same call). - Waits for a few words (e.g., a **hit after three words**) rather than matching every word. - On a **cache miss** (e.g., after the common prefix "you said your name is" when a unique name arrives), cached audio is **yielded out** while remaining text streams in. - The user perceives this as very fast response times. - **Cartesia integration (stated):** - Agent stream is piped through both the cache and a **Cartesia websocket**. - Cartesia "doesn't know about the existence of this prefix cache" and generates the full sentence with natural prosody. - Since cached audio has already played, Cartesia's audio for the cached portion is **suppressed**, and only the remainder is played. - **Qualification (Bo):** "There might be a tiny bit of a hiccup... you probably won't be able to notice," so the result "looks seamless to the user." --- ## Demonstration Call - Bo plays a recorded call: a patient calls "Bo OBGYN" (Elise) to schedule an ultrasound; the agent collects name, date of birth, new-patient status, texts an insurance link, and books an appointment. - **Claim:** The call shows the streaming and background machinery in action; "a lot of things are happening in the background... this is what really makes voice agents interesting," and "a lot of effort... in the harness" is needed to achieve natural conversation. --- ## Closing: About Elise - **Claims (Bo):** - Elise is headquartered in New York and is expanding its presence in the Bay Area. - It is a "different style of company" than typical SF AI startups — focused on "helping people... in life's most critical areas," specifically **housing and healthcare**. - The company is "doing really well"; recruiting via a join link and posting on Twitter (@EliseAI). --- ## Unresolved Tensions / Qualifications Preserved - **Affiliation discrepancy:** Speaker says "we at Xnor.ai architected our voice agent harness"; title metadata says EliseAI. Not resolved by transcript evidence. - **Audio seam:** Bo acknowledges a potential "tiny bit of a hiccup" at the cache/handoff boundary while asserting users likely won't notice it — both the imperfection and the mitigation claim are preserved. - **Speaker self-description:** He credits the demonstration voice as "Elise Trial" (patient) and "Bo OBGYN Elise" (agent) in the recorded call — demo content, not a claim about real persons. # Speaker Attribution Record ## Video **Title:** Realtime Voice Agents with Frontier Intelligence — Bohan Li, EliseAI **Video ID:** `01a15ca7-61ee-7e5e-bfff-b162f87c6e42` ## Speakers & Roles ### Primary Speaker - **Name:** Bohan Li (introduces himself as "Bo") - **Role:** Creator / presenter - **Affiliation (self-identified):** EliseAI (also references prior work at **Xnor.ai**) - **Speaking segments:** Full technical presentation on voice agent architecture, plus a closing segment about EliseAI ### Demo Recording Voices (within played call recording) The recorded demo call contains two synthetic/agent-user voices, not live speakers: - **Voice agent:** "Bo OBGYN Elise" (synthetic agent in demo) - **Caller (user):** "Elise Trial" (demo caller persona, recorded) ### Other - **Audience:** `[applause]` at end — no speaking role ## Segment Structure | Segment | Speaker | Content | |---|---|---| | Technical presentation | Bohan Li | Cascaded voice agents, streaming speculative transcriber, LLM layer, background tool calling, TTS prefix cache | | Recorded demo call | Agent & caller (recorded) | OBGYN appointment scheduling scenario | | Closing segment | Bohan Li | About EliseAI | ## Promotional Segments — Flagged - **Company promotion (closing segment):** Bohan Li describes EliseAI's mission (housing, healthcare), New York headquarters, Bay Area expansion, hiring, and links to join the team and follow **@EliseAI on Twitter**. ## Referenced Sources & Third-Party Technologies (as claimed by speaker) - **Cartesia** — text-to-speech engine with WebSocket support - **Flux** — fast streaming transcriber - **Scribe V2** — accurate batch transcription layer (attribution not stated as first-party vs. third-party) ## Third-Party Claims - Speaker claims prior work in **self-driving cars** (personal background, unverifiable from transcript) - Claims about EliseAI company performance ("we're doing really well") — self-reported, no citation ## Availability & Provenance Limits - **Curated metadata:** None supplied beyond title - **Comment availability:** Not indicated - **Timestamps:** Not present in transcript; segment boundaries approximate - Speaker name in title ("Bohan Li") vs. self-introduction ("Bo") — presumed same person; identity beyond this is unverified \>\> My name is Bo. I'm going to be here presenting real-time voice agents with Frontier Intelligence. Effectively, going to be talking a little bit about how we at Xnor.ai architected our voice agent harness to get real-time voice with the Frontier level of intelligence that we need. Okay. So, before I start, I think I wanted to kind of draw some parallels about why we decided to go with cascaded voice agents and especially kind of comparing that to self-driving cars which I was working in before. So, to me, cascaded voice agents makes a lot of sense when you view it in lens of kind of breaking it down into perception which is for self-driving cars, it's you know, the bounding boxes, the camera, the lidar. For voice, it's going to be the transcription. Basically, effectively turning these like signals from the real world into elements of data that the language model or whatever brain you're working on can process. Second one is the planning step which is pretty straightforward. This is where the language model will take in the outputs from the perception stage and produce the outputs that you want to produce out back and out into the real world. And finally, there's the controls layer where on self-driving, you'd be taking the trajectory that the planner would output and kind of turn it into the real controls to kind of build like drive the car. Here, we're turning the text into audio that we use to express our voice agent's thoughts. And um yeah, so here I'm going to be like going to diving into each one of these elements and we've made a few kind of interesting tricks on each of these areas to improve the speed of our voice agents without sacrificing the intelligence. So, the first one is going to be uh the transcriber layer. So, we came up with this concept called like the streaming speculative transcriber where effectively we are layering a fast streaming transcriber like Flux on top of or kind of below a uh scribe V2 or a accurate batch transcription which kind of takes in more context. It's a little bit slower, but it will give you more accurate detections. So, we're going to walk through a scenario. So, in this in this case the agent just asked, you know, providing can you provide your name and date of birth and the user is going to say this and we'll see how that plays out um timing-wise. So, first we're going to get, you know, the short detection. Um we'll get it from we'll get it from the streaming layer. The accurate layer uh the corrective layer is not going to fire because it's the same text. Um we're going to get some more streaming text detections and in this case the corrective layer is actually canceled because we got new um new text. So, you know, more context, more audio is going to beat the old accurate one. And here's where kind of the first correction comes in. So, because the scribe V2 layer understands, you know, the the context of the question, it's able to understand that this is talking about name and this is a date of birth. Then a couple more detections, these are just punctuation, we don't care. And so, in the end we kind of release this text over to the agent. And moving on um to the language model layer. So, here since we're kind of using these slow but intelligent LLMs, we really want to reduce the number of round trips and the thing that causes us to do a lot of inferences is tool calling. So, one way to get rid of that is by having background agents do the tool calling for you and kind of um push the tools back into the context of the main agent so that it thinks it made the tool call, but um but it it it really didn't. So, uh so, we remember from like detections from before. So, well, what happened is each one of these detections is going to trigger a um an early kind of generation of the agent and we but we won't actually emit this out until we're confirming that the user has finished speaking. So, in this case, the user says, "Sure." The agent kind of knows that the user is about to say something else. Our background tool calling here, which is going to be helping us find figure out the name and the date of birth from the user detection, is not firing. So, nothing much there. Um the next instant detection comes in. It says that, you know, still not really a name. Um our agent kind of plays along and continues there. Now, kind of a more more context come comes back. The agent kind of feels like there should be a name. It's going to ask to spell it out because it's probably thinking there's some transcription error here. Still no name or date of birth. And then finally, this you remember this is kind of our corrected um final instant detection from the transcriber from the Scribe V2. Um here, our eager kind of agent generation that was made without any tool calls is going to get canceled because the background agent finally is able to find the name and date of birth it's looking for. So, it's going to retrigger and now the the agent actually has the context it needs. Um and you see here, it's kind of we're doing it the tool call here is a little bit um um some intelligence there. We're going to be like, you know, correcting mis-transcriptions of name, and doing some like phonetic matching here. Um and yeah, and then we'll kind of once we've understood that this is the end of the user utterance, we'll kind of emit it out. So, pretty standard. Okay, and then the next layer here is going to be text-to-speech. So, with text-to-speech the goal is to kind of take what the agent said, and the agent's going to be emitting this in a streaming fashion. So, we're going to need to um produce audio as quickly as possible. And ideally, what you can do is before the agent has even finished generating the full text you can have the audio play, so it's kind of hiding the latency of finishing the generation. So, um I'm going to kind of play the streaming um the stream the streaming uh agent output now. So, starts with you. And yeah, actually before I uh further, there's this new concept that we're introducing here called the prefix cache. So, the prefix cache is going to be looking at the um agent stream, and seeing if we already have generated audio for that sequence of words um from like a prior generation, or maybe like the same generation um in this uh in this call as as well. So, um it sees the word you. Uh we for this prefix cache, we're going to be, you know, we don't want to like immediately hit on every single word. We're going to be waiting for a little bit more words. Um so, after three words, the prefix cache gets our first hit. And um over here on the right, this is kind of our text-to-speech standard provider, you know, Cartesia is a text-to-speech engine with web socket support. So, we're we're piping the agent through the the cache, and also piping it through web socket. Um more tokens come in, more cache, more sending through web socket. Not much to say here. And okay, so now we get our first uh kind of first unique thing, which is we found the token that actually causes a cache miss. And it makes sense. If we're kind of caching previous generations, um you said your name is is a pretty common thing, but once you we add in the name, suddenly we're that's that's going to result in the cache miss. At this point, we're actually going to yield out our cached audio. So, you said your name is is going to be um emitted as the rest of the streaming text is coming back. So, at this point, the user hears the agent and user doesn't really know what's going on. They just looks like really fast response times to them. Um and now the kind of remaining text flows through. And at this point, we've already emitted from the cache. The cache has done its job. Um the rest we can kind of throw into Cartesia. And here's kind of the trick where Cartesia has seen the entire transcript up to this point. It to to Cartesia, like it doesn't know about the existence of this prefix cache. It's just going to generate this full sentence with, you know, standard natural prosody. But, what we do is when the generation comes back, since we've already played the audio here, we can actually suppress the audio from Cartesia here and just play out the remaining stuff. So, the user, there's might be a tiny bit of a hiccup. You know, I'll play some audio later and you'll know that you probably won't be able to notice. And effectively, we're kind of taking this and or taking this part and just emitting it directly after this audio has finished playing. So, it looks seamless to the user. And yeah, so I guess at this point kind of this is a lot of technical details. I'll I mean the best way is just kind of play a call and see see this in action. So here I recorded a call and let's just hear what happens. \>\> This is Bo OBGYN Elise speaking. \>\> Hi, I think I might be pregnant and I want to schedule an ultrasound to confirm. \>\> That's exciting news. Can you provide your name and date of birth? \>\> Sure, it's Elise Trial. Date of birth 2303. \>\> Thank you. Are you a new patient with us? \>\> Yes. \>\> Welcome to Bo OBGYN. Can I text you a link to upload your insurance information? \>\> Sure. \>\> Okay, sent. Let me know when you've received it. I've received your insurance information. The earliest appointment I have is Thursday, July 2nd at 10:00 a.m. \>\> Actually, give me a second. Let me check my calendar. \>\> Sure, take your time. \>\> Um do you have something for next week? \>\> For next week, I can offer an ultrasound with Dr. Avery Stone at North Clinic on Tuesday, July 7th at 2:00 p.m. or 3:00 p.m. Do either of those work for you? \>\> Yeah, 2:00 p.m. works. \>\> Great. Your appointment has been booked. We look forward to seeing you then. \>\> Thanks. Bye-bye. \>\> All right. Yeah, that's pretty much it. Um Yeah, you can kind of see our all this like uh streaming and you know, a lot of things are happening in the background and and you know, this is what really makes like voice agents interesting. And um there's a lot of effort that can be done in the harness to really kind of get a um natural conversation, which is what we're after. Uh okay. Yeah, so I guess briefly, you know, in the last part, I want to just talk a little bit about Elise. So I think Elise, you know, our headquarters are in New York and kind of we're trying to expand our presence here in the Bay Area. Um we I think it's maybe like a different style of company that I think people are like uh think of when they think about AI startups in San Francisco. Where we're actually very focused on um just like helping people and helping people where they need it, like kind of the life's most critical areas. We work on housing, health care and uh we're doing really well and you know, here's there's a link here to um you kind of join our team and there's going to we're going to be uh posting a lot on Twitter, so you can follow us at EliseAI as well. Um yeah, that's that's it. \>\> \[applause\]