A roughly seven-minute talk by Ornella Bahidika (Microsoft) and Joel Allou on keeping control of multi-step flow agents outside the LLM, illustrated by Ace, a voice tutor in production. The lesson is modeled as a small state machine (intro, teach, check, grade, advance, wrap) where each node gives the LLM a narrow contract; an external harness validates returns, advances state, decides next steps, and overrides model attempts to skip ahead or declare completion prematurely. Covers the contract shape, shared canvas and interruption handling, cases where the speakers gave the LLM more authority and rolled it back, and a list of decisions the LLM should never own in a flow agent. # Video Identity Observations ## Presenters & Affiliation - Presenters introduce themselves as **Ornella Bahidika and Joel Allou** (title metadata), associated with **Microsoft** per the title. - They identify as builders of a product: *"we built Ace, a live AI voice tutor that runs a full lesson start to finish reliably."* ## Explicit Self-Description - **Product claim:** Ace is a "live AI voice tutor" that speaks "back and forth with students." - **Core thesis of the talk:** *"The trick is LLM is not in charge"* — reliability is a **control problem, not a prompting problem**. - Framing metaphor offered explicitly: *"The model is the talent, and the harness is the director."* ## Intended Audience - **Practitioners who ship AI agents**: *"If you have shipped a multi-step agent, you know this moment"* — assumes experience with agent failures (skipped steps, loops, premature completion). - Developers concerned with **reliability, cost, and latency**: they describe needing "something that is reliable, something that is cost-effective, and something that is fast." ## Claimed Scope of Content - **Harness/state-machine engineering:** lessons structured as "a small state machine with intro, teach, check, grade, advance, and wrap," where "the model never decide where we are." - **Model-size economics:** by confining the model per step, they "were actually able to rely on something like a Haiku 4.5, which is a much smaller model... saving money, saving time, and saving latency." - **Generalizability claim:** the pattern is "applicable to... a voice model... coding agents, Ops Run Books... onboarding flows." - **Heuristic offered:** *"If it's somewhat of a coin flip, then you want to take the control flow out of the model."* ## Observable Presentation Patterns - **Demonstration with logs:** they play a recording showing "logs on all of the different harnessing that are happening" (section inputs, whiteboard drawing, queue clearing, lesson ending). - **Concrete model references:** mentions "Opus 4.7 Cloud from Anthropic" and "Haiku 4.5" by name. - **Closing tagline consistent with title:** *"Don't let the model talk... actually let it talk, but don't let it drive."* - Talk format suggests a **conference/demo session** (closing "Thank you" from an unseen party). ## Claims vs. Demonstrated Evidence - **Creator claims:** reliability, cost savings, and latency gains from harness design — asserted, supported by a log-recording demo but no quantitative metrics shown in transcript. - **Demonstrated:** a structured architectural argument with a state-machine breakdown, a demo walkthrough, and a decision heuristic for when to apply the pattern. # Delivery Observations: "Don't Let the LLM Drive" ## Speakers & Structure - **Two-person tag-team format**: Ornella opens with the framing, then hands off explicitly: *"Joel is going to show you the harness thing."* Joel carries the technical middle section and closes. - Opening self-introduction in plain style: *"Hi, I'm Onela. That's Joel."* ## Tone - **Conversational and confident**, with light humor: - *"Let it talk, but don't let it drive."* (title callback in the closing) - *"The demo never show you that."* - **Empathetic opener** targeting a shared pain point: *"If you have shipped a multi-step agent, you know this moment."* ## Extended Metaphors - **Theatre metaphor** used repeatedly by Ornella: - *"The model is the talent, and the harness is the director."* - *"The model is brilliant at delivering a line, but it's really terrible at remembering if it's on step three of six."* - **Contract metaphor**: *"Each step sends the model a neural contract. Do this one thing, return it."* ## Recurring Phrases - **"Right?"** — Joel's most frequent verbal tic, appearing throughout (e.g., *"For something like this… Right?"*, *"…in between, right?"*, *"…engineered that outside of the model, right?"*). Appears **15+ times**; clearly a stable habit, not incidental. - **"So, …"** — sentence-starter used repeatedly by both speakers (*"So we stop asking it to."*, *"So, when we think about the frontier models…"*, *"So, by doing this…"*). - **"Again"** as a discourse marker by Joel: *"And so, the model never really… It proposes… again… we have engineered that outside of the model, right? So, again…"* ## Teaching Habits - **Repetition for emphasis** rather than new information: - *"the model never decide where we are"* (Ornella) → *"the model never really has to think"* (Joel). - **Concrete enumeration**: Joel lists the lesson state machine steps — *"intro, teach, check, grade, advance, and wrap"* — and later *"three things"*: when the lesson is done, whether the student got it right, and what comes next. - **Live demo walkthrough**: *"let's go ahead and play this recording, which will show us logs… as you can see in this video, especially on the right side…"* - **Specific named examples** to ground abstractions: *"Opus 4.7 Cloud from Anthropic"*, *"Haiku 4.5"*. - **Generalization beyond the product**: *"It's applicable to coding agents, to Ops Run Books… onboarding flows."* - **Memorable decision heuristic** offered near the end: *"If it's somewhat of a coin flip, then you want to take the control flow out of the model."* ## Filler & Disfluencies - Occasional **"um"** and hesitations from Joel (*"um Opus 4.7"*, *"the the this is applicable"*), plus a **[clears throat]** during the demo — suggesting partially improvised delivery. - Self-correction at the close: *"don't let the model talk, right? Or actually let it talk, but don't let it drive."* ## Uncertainty Note - Transcript lacks video/audio, so pace, gestures, and slide cues are unobservable; filler frequency may be under- or over-represented by transcription quality (e.g., "his A" likely a mis-transcription of "Ace"). # Position Record: "Don't Let the LLM Drive" — Ornella Bahidika & Joel Allou (Microsoft) ## Central Claim **The LLM should not hold control flow.** Reliability in multi-step agents is a **control problem, not a prompting problem**. The model should execute narrowly scoped tasks; a deterministic harness should direct the process. ## Ornella Bahidika's Positions - **Product**: Built **Ace**, a live AI voice tutor running full lessons start-to-finish reliably; "the trick is the LLM is not in charge." - **Failure pattern**: Multi-step agents near demo succeed, then fail with real users — ending early, skipping steps, or looping. Demos don't reveal this. - **Common but wrong fix**: "Prompt it harder, add more holes [guardrails]" — rejected because "reliability was never a prompting problem." - **Design analogy**: *"The model is the talent, and the harness is the director."* The model is brilliant at delivering a line but terrible at tracking position (e.g., "step three of six") — so don't ask it to. - **Architecture**: A lesson is a **small state machine**: `intro, teach, check, grade, advance, wrap`. Each step sends the model a "neural contract" — do one thing, return it. The harness validates output, advances state, decides what's next. **The model never decides where we are.** ## Joel Allou's Positions - **Critique of frontier-model usage**: People leverage models like **Claude Opus 4.7** "for everything — thinking, processing, and everything in between." He allows this "can be good" but is "not always effective" for a live voice tutor needing **reliability, cost-effectiveness, and speed**. - **Harness engineering**: Build the steps; provide only the input needed for each specific scenario so the model is **confined to one action in one step**. - **Result**: Instead of a heavy model, they run **Haiku 4.5** — smaller, less reasoning capability — but harnessing lets it "perform at the level we expect, saving money, saving time, and saving latency." - **Demo/logging evidence** (claimed, shown in recording): logs on the right side show harnessing for sections, whiteboard drawing, clearing the queue, and lesson-ending steps; new scenarios are incorporated into the state machine. - **Division of labor**: *"The model never really has to think. It proposes, but ultimately it is the harness that decides."* - **Three engineered decision categories for Ace**: (1) when the lesson is done, (2) whether the student actually learned, (3) what comes next — all decided **outside the model**. - **Generality**: The approach applies to voice models, coding agents, ops runbooks, and onboarding flows. - **Heuristic for adoption**: If agent reliability is "somewhat of a coin flip," take the control flow out of the model; build decisions around it and feed it easy inputs. ## Closing Summary (both) > "Don't let the model talk — actually let it talk, but don't let it drive." ## Qualifications & Tensions Preserved - Joel concedes full-model delegation "can be good" — his objection is situational, not absolute. - Model naming ("Opus 4.7 Cloud," "Haiku 4.5") is as stated in the transcript; efficacy claims (matching expected performance, latency/cost savings) are speaker assertions without external evidence in the source. - The effectiveness claims rest on a demo recording of logs; no quantitative results are presented. # Video Attribution Observation ## Video Metadata - **Title:** Don't Let the LLM Drive - Ornella Bahidika & Joel Allou, Microsoft - **Video ID:** `01a15ca8-722b-7ab8-ac8a-f03da3cd95a3` - **Type:** Conference/demo presentation on AI agent architecture ("harness engineering") ## Speaker Roles | Speaker | Role | Basis | |---|---|---| | **Ornella ("Onela") Bahidika** | Co-creator / co-presenter | Self-identified: "Hi, I'm Onela... we built Ace" | | **Joel Allou** | Co-creator / co-presenter | Identified by Ornella: "That's Joel"; takes over with "Yeah. So, when we think about the frontier models..." | | **Unidentified speaker** | Off-screen/closing voice | Utters final "Thank you" — role unstated | - Both presenters identify as **Microsoft** employees (per title metadata, not self-stated in transcript). ## Provenance & Referenced Items ### Products / Systems Mentioned - **Ace** — live AI voice tutor built by the presenters; described in a recorded demo with logs (whiteboard drawing, queue clearing, lesson-end steps). ### Third-Party Models Cited - **Anthropic Opus 4.7 Cloud** (transcript wording) — cited as example of a frontier model. - **Haiku 4.5** — smaller model they report using instead, due to harnessing. ### Claims About Third Parties - Anthropic models' capabilities characterized by the speakers; **not independently attributed** to Anthropic. ## Promotional Segments - The Ace demo recording is a **self-promotional segment** showcasing the presenters' own product; flagged as such. ## Uncertainty Notes - Transcript renders the name as "Onela"; title metadata says "Ornella." - Speaker changes between Ornella and Joel are inferred from self-introductions and hand-off phrasing; no explicit per-utterance speaker labels exist in the transcript. - **Comment availability:** not indicated in supplied material. - No external sources, citations, or references beyond those listed were present. Hi, I'm Onela. That's Joel. And we built Ace, a live AI voice tutor that runs a full lesson start to finish reliably. The trick is LLM is not in charge. If you have shipped a multi-step agent, you know this moment. It's near the demo, then a real user gets in and halfway through the agent decide it's done. Or skip a step, or even loops. The demo never show you that. And the first fix everyone reaches for is prompt this harder, add more holes. But reliability was never a prompting problem. It's a control problem. Think of it this way. The model is the talent, and the harness is the director. The model is brilliant at delivering a line, but it's really terrible at remembering if it's on step three of six. So we stop asking it to. A lesson is a small state machine with intro, teach, check, grade, advance, and wrap. Each step sends the model a neural contract. Do this one thing, return it. The harness validates what's comes back, advance the state, and decide what's next. The model never decide where we are. That's the design. Joel is going to show you the harness thing. \>\> Yeah. So, when we think about the frontier models of today, let's take for example um Opus 4.7 Cloud from Anthropic, you see that oftentimes people leverage the model for essentially everything. For the thinking, for the processing, right? And for everything in between. While that can be good, it's not always effective in situations like ours, where we are building a live AI tutor that is speaking back and forth with students. Right? For something like this, we had a need to actually build something that is reliable, something that is cost-effective, and something that is fast. Right? So, this is where the idea of leveraging the concept of harness engineering has come in. Where instead of having a model that is really intelligent, sort of go through everything for us, we will build all of these steps that are needed and provide only the input required for the model to execute a specific scenario. So, when we were building ACE, we actually thought very deeply about state machines. Right? What is the step right now? And what is the possible steps that could come after? And within each of the steps, what are concrete things that we can provide to the model so that it is confined to that specific action, it is confined to that specific step at that particular moment, and only execute what needs to be done. So, by doing this, instead of having a very heavy model like a 4.7, we were actually able to rely on something like a Haiku 4.5, which is a much smaller model, doesn't have as much reasoning capabilities, but because of the harnessing around it, it's still able to perform at the level in which we expect, saving money, saving time, and saving latency. So, let's go ahead and play this recording, which will show us logs um about a particular lesson. So, as you can see in this video, especially on the right side, we see logs on all of the different harnessing that are happening. \[clears throat\] Right? So, for example, we see that there's harnessing for a section, which provides input to the model about exactly what to speak about, what to do. We have harnessing about drawing on the whiteboard. We have harnessing that deals with clearing the queue. We have steps to how to end the lesson and everything in between, right? So, everything that would allow us to actually build the lesson in a way that is reliable even if there's a new scenario that comes in. We try to incorporate that in our state machine. We try to incorporate that with within the lesson, right? So, again, the model all it worries about is given an input, it knows which action to take and it provides the output of that action. Right? And so, the model never really um has to think. It proposes, but ultimately it is the harness that decides. And so, for his A specifically, there are three things that we wanted to think about. Like, when is the lesson done is one, right? Did the student actually get it right? Like, did they actually learn in the way they were supposed to? And what comes next, right? And so, everything that comes within those three categories, all of the different questions, all of the different actions that the models needs to take, we have engineered that outside of the model, right? So, again, it's an input, the model receives it and gives us an output, right? And so, this is very, very important and we have found this to be very remarkable. Right? So, again, the the this is applicable to really everything, right? It's applicable to something like A that is a voice model. It's applicable to coding agents, to Ops Run Books. Um it's applicable to onboarding flows, right? The same rule applies, right? We want to find a way to not let the model think, but building abstractions around it. So, a good way to to remember on whether you should use this abstraction is essentially to think about the reliability of your agent, right? If it's somewhat of a coin flip, then you want to take the control flow out of the model. You want the model to not make as many decisions as it should and instead build those decisions around the model and simply feed an easy input so that the model can easily produce an output, right? So, don't let the model talk, right? Or actually let it talk, but don't let it drive. So, we're Joel and Ornella. This is Ace, and if you have any questions, please let us know. Thank you. \>\> Thank you.