Talk by Vinoth Govindarajan (OpenAI) on agent harness failures, framed around the OpenClaw incident where a second write silently erased a first write in a shared session while the agent answered confidently from stale state. Covers harness reliability topics: state persistence, single writer lanes, overlapping writes, idempotency, locks, ordering, deadlines and cancellation, approval drift and bounded authority, and user-visible proof of delivery. Core thesis: the model proposes, the harness commits, and a receipt proves it. Concludes with a run receipt audit of five questions per incident: what woke the agent, what state it inherited, what authority it used, what executed, and what evidence survived. # Video Identity Observations ## Speaker Self-Description The speaker introduces himself mid-talk, though the transcript contains an apparent inconsistency between the stated name and the video title: - The video title credits **"Vinoth Govindarajan, OpenAI"**. - The transcript self-introduction states: > "Hi, I'm Ben. I work on core data and AI infrastructure at OpenAI. Before that, I worked on distributed systems at Apple and Uber." - He also describes a side project: > "I write the agent stack where I try to explain how the production AI and data systems work under the hood." - An additional confusing line appears: > "I'm daughter of OpenClaw and this is not a openclaw product pitch" — likely a transcription artifact; the speaker explicitly disclaims a product pitch. ## Claimed Scope and Purpose - **Central thesis**: > "I want to convince you all that most of the agent failures are not model failures. Those are harness failures." - Framed as a **"pure system design talk"** using OpenClaw as a **"public case study because its issues code docs makes the harness around the agent unusually visible."** - Explicit takeaway contract: > "A model proposes, the harness commits, and the receipt proves it." - Audience call to action: a five-question audit ("What woke it up? What state did it inherit? Which authority did it use? What executed? What evidence survived?") framed as something to "run when you get back to your team." ## Intended Audience (Demonstrated) - **Engineers building or operating production agent systems**, evidenced by: - Technical depth on state ownership, commit ordering, idempotency, locks, watchdogs, and audit receipts. - References to specific agent products: OpenClaw, "Hermes," Codex, Cursor, "open code," and Claude Code. - Closing pointers to the **OpenAI Agents SDK** and his own writing, plus: > "I'll be at the open booth after this talk if you want to talk about your harness design" — indicating a **conference talk** setting. ## Observable Presentation Patterns - **Structured pedagogical style**: numbered failure modes (state hole, overlapping writers, dangling tool call, approval drift, missing edge proof), a three-point mnemonic ("Own the state, order the mutation, prove the action"), and a car analogy (model = engine, harness = steering/brakes). - **Case-study driven**: each failure mode is anchored to a described OpenClaw GitHub issue. - **Cross-company credibility signaling**: prior work at Apple and Uber on distributed systems is cited as background. - **Promotional (mild) closing**: QR codes pointing to the OpenAI Agents SDK ("all these harness are already built in") and the speaker's newsletter/site "the agent stack." ## Claims vs. Demonstrated - *Creator claim*: failures are harness failures, not model failures — argued, not proven within the talk. - *Demonstrated*: deep familiarity with distributed-systems failure modes applied to agent architecture; conference-style presentation with audience-directed exercises. - No channel-wide description (e.g., subscriber counts, upload cadence, channel name) is present in the material. # Delivery Observations — "The Model Was Right. The Harness Failed." ## Tone - **Plain, declarative, and assertive.** The speaker states claims crisply and repeats them as rules rather than suggestions. - "My goal today is simple." - "Open clause is the case study and the contract is the takeaway." - **Slightly wry and aphoristic**, especially in contrasts: - "A powerful engine with no brakes is not autonomy. It is a liability with good acceleration." - "Silent success gives you a lie." - "Bound the work before the work bounds you." ## Structure & Teaching Habits - **Announces a thesis up front, then bookends the talk with it.** The talk opens and closes on the same three-part rule. - Opening: "I want to convince you all that... agent failures are not model failures. Those are harness failures." - Closing: "Let me recap the same three things I asked you to remember... Own the state, order the mutation and prove the action." - **Explicit memory instruction to the audience** — a stable rhetorical device: - "If you remember only three things from this talk, make it these." - **Incident-first teaching.** Begins with a concrete production failure before credentials or framework: "So let's start with one production incident." - **Numbered audits and repeatable question templates.** Delivers checklists the audience can reuse: - "The audit has five questions. What woke it up? What state did it inherit? Which authority did it use? What executed? And what evidence survived?" - "For each one, let's ask the same question. What did the user see? Which boundary it broke? And what would the receipt have caught?" - **Analogies to explain architecture.** The car analogy is introduced explicitly as a mental model: - "The model is the engine... But nobody buys a production car by just looking at the horsepower alone." - "The model gives you capability, but the harness gives you control." ## Recurring Phrases & Vocabulary - **A small set of terms is repeated nearly mantra-like:** - **"Harness"** and **"model"** as paired opposites throughout ("The model proposes the harness commits and the receipts proves it" — appears near the start and again near the end). - **"Receipt"** as a keyword concept: "A transcript tells you what the agent said. A receipt tells you what the system allowed, attempted, executed." - **"Boundary"** used constantly: "The model is not the production boundary," "The missing boundary was not intelligence," "Every external boundary needs an ending." - **Contrast-pair construction** is the dominant rhetorical pattern: - "Capability is not execution." - "Internal success is not external proof." - "Storage tells you where the bytes live. Ownership tells you who can reconstruct the reality." - "Proof is a chain, not a claim." - **Distributed-systems vocabulary** carried over deliberately: "timeouts, retries, item potency, locks, ordering, and state ownership," "last writer wins is not a consistency model," "one ordered commit path." ## Communication Patterns - **Reframing user-visible symptoms as system properties:** - "Users do not see cues or locks. They see behavior." - "Ordering is a product feature because users experience ordering books as personalities." - **Direct, actionable homework assigned to the audience:** - "Pick one agent system, not all of them. One trace one production, one real production path and ask for the receipt." - **Closing with availability and self-reference to his own writing:** - "I write the agent stack where I try to explain how the production AI and data systems work under the hood." - "I'll be at the open booth after this talk if you want to talk about your harness design." ## Disfluencies & Transcript Artifacts - Frequent **stalls and restarts** ("uh," repeated words): "Let me recap all the incidents. Here are the file failure shapes you you to look for." This suggests conversational delivery rather than read-from-script. - Occasional **throat clear** mid-explanation: "[clears throat]" at [320]. - Some transcript garbling (likely speech-to-text errors), e.g., "I'm daughter of OpenClaw," "reconcent the fact," "expose squashalty" — delivery itself likely clearer than the transcript implies. **Uncertainty noted:** these specific wordings reflect transcription, not confidently observable speech. ## Stable vs. Isolated - **Stable patterns** (repeated, well-evidenced): contrast-pair phrasing, "harness vs. model" framing, receipt/boundary vocabulary, checklist-style recaps, thesis bookending. - **Isolated moments:** the car analogy (used once), the self-introduction block, and the QR-code close — all one-off structural beats, not recurring habits. # Position Record: "The Model Was Right. The Harness Failed." — Vinoth Govindarajan (OpenAI) ## Speaker Note The transcript contains a self-introduction as "Ben" while the video metadata identifies the speaker as Vinoth Govindarajan; the transcript also says "I'm daughter of OpenClaw" (likely "founder/author of agent stack" garbled). All claims below are attributed to the speaker as identified in metadata. ## Central Thesis - **Most agent failures in production are harness failures, not model failures.** - Production contract: **"A model proposes, the harness commits, and the receipt proves it."** The model may suggest a message/tool/edit/command, but it is not the production boundary; the harness owns state transition, authority checks, ordered commits, and receipts. ## Opening Incident: Silent Success - User asked the agent to remember a refund; the assistant confirmed, but the next turn could not recall the fact. - The user-visible edge looked healthy while the durable record had a hole — **not** a hallucination, crash, or bad answer. - Speaker's reasoning: a crash at least gives a boundary and recovery point; **"silent success gives you a lie"** — delivery can succeed while persistence fails, and the model stays coherent over broken history. ## Core Takeaways (Speaker's Request) 1. **Own the state** — one owner, one replay path per fact. 2. **Order the mutation** — one ordered commit path for shared mutable state. 3. **Prove the action** — a transcript shows what the agent *said*; a receipt shows what the system *allowed, attempted, executed*, and what the user-visible edge confirmed. ## Car Analogy - Model = engine; harness = steering, brakes, road rules, dashboard, black box. - **"A powerful engine with no brakes is not autonomy. It is a liability with good acceleration."** ## Harness Blueprint - Shared across agents (OpenClaw, Hermes, Codex, Cursor, OpenCode, Claude Code, per speaker): - Events enter from many surfaces (chat, webhook, timer, heartbeat, external systems). - Control plane maps events to a **session key** defining the state boundary. - Session lane provides one active writer; runtime calls models/tools; tools act through approvals and policies; audit trail becomes the run receipt. - The runtime is stateless; the harness rebuilds working state per turn. **Coherence does not prove the working set was complete.** - These failure modes (timeouts, retries, idempotency, locks, ordering, state ownership) are not new — agents make them easier to trigger and harder to explain. ## Five Failure Shapes ### 1. State Hole (delivered but not remembered) - OpenClaw issue: a Telegram reply succeeded while the turn was not written to active context/transcript; next turn lacked the durable record. - **Ownership ≠ storage**: storage says where bytes live; ownership says who can reconstruct reality (calendar events → calendar system, turns → session transcript, preferences → memory store). - Test: for every fact, who owns it and how would you replay it? No replayable owner ⇒ not reliably remembered. ### 2. Overlapping Writers - OpenClaw issue: load-modify-save race; second save silently erases the first. Both operations locally correct. - **"Last writer wins is not a consistency model."** - Rule is not "no concurrency" (too slow): fan-out subagents, parallel reads, concurrent sessions are fine. Rule: **one ordered commit path per mutable state boundary** (queue, mutex, transaction, or lock). - Ordering is a product feature — users experience ordering bugs as personality flaws (forgetful, dead, confused). ### 3. Dangling Tool Call (lifecycle) - Session contains a tool call with no matching tool result (dead process, dropped connection, timeout). Run waits forever; new messages queue behind silence. - Fix: deadlines, cancellation, watchdogs, tool timeouts with error results, recovery channels that don't wait behind stuck work. - **"Bound the work before the work bounds you."** ### 4. Approval Drift (authority) - OpenClaw issue: an expired approval callback treated as retriable; stale callback state blocked later channel work. The button click existed; valid authority did not. - Approval is a **scoped execution state**, not "vague memory that a human clicked yes." It must bind to action, session, run, tool, arguments, duration, outcome, and point to the receipt. - **"Capability is not execution... The model can request, but the system decides."** ### 5. Missing Edge Proof - OpenClaw issue: message tool reported success but message never rendered; normal assistant reply still appeared. Agent later claims "I already sent it"; user truthfully says "I never saw it." - **Internal success is not external proof; proof is a chain, not a claim** (model proposes → policy allows → execution attempts → user-visible edge confirms). ## Prescribed Audit (Five Questions per Production Trace) 1. **What woke it up?** — trigger and identity (message, webhook, timer, subagent, replay). 2. **Which state did it inherit?** — transcript, session state, memory, policy version, tool surface. 3. **Which authority did it use?** — actor, session, tool, run, arguments, scope, lifetime. 4. **What executed?** — tool/API calls, arguments, attempt number, idempotency key, external results (side-effect boundary, not the model's summary of intent). 5. **What evidence survived?** — receipt should end at the boundary the user cares about. ## Closing Position - Applied to the opening incident: trigger = user message; state = the broken boundary; execution = channel send; delivery survived while the durable turn did not. - The system needed a better harness, not a better model or prompt. - Once text can become action, the question shifts from "can the model reason?" to "can the system own state, order mutation, bound work, constrain authority, and preserve evidence?" - References: OpenAI Agents SDK (harnesses built in) and the speaker's writing on production agent systems; the speaker notes this is a systems-design talk, explicitly **not** an OpenClaw product pitch — OpenClaw is used as a public case study because its issue tracker makes the harness unusually visible. # Video Attribution & Provenance Record ## Video Identification - **Title:** The Model Was Right. The Harness Failed. — Vinoth Govindarajan, OpenAI - **Video ID:** `01a15ca8-3445-7ac9-a324-f4cd124609d7` - **Note:** The title names Vinoth Govindarajan, but the transcript's self-introduction gives the speaker's name as "Ben" — this discrepancy is unresolved from the source material. ## Speaker Roles ### Primary (and only) speaker: "Ben" — Presenter/Creator Self-introduction from transcript: > "Hi, I'm Ben. I work on core data and AI infrastructure at OpenAI. Before that, I worked on distributed systems at Apple and Uber." - **Role:** Talk presenter, sole speaker throughout. - **Affiliations stated by speaker:** OpenAI (current); Apple and Uber (prior, distributed systems). - **Personal project:** Author of "the agent stack," a writing project on production AI/data systems. ### No other speakers - No guests, narrators, or audience speakers appear in the transcript. ## Promotional Segments (Flagged) 1. **OpenClaw case study disclaimer (explicit):** > "This is not a openclaw product pitch. This is a pure system design talk. I'm using open claw as a public case study because its issues code docs makes the harness around the agent unusually visible." 2. **Closing promotion (explicit):** > "The first points to the agent open AI agents SDK where all these um harness are already built in so that you can use to build your own agents and second points to the agent stack where I write about the production agents systems in more detail. I'll be at the open booth after this talk..." - Promotes the **OpenAI Agents SDK** and the speaker's **"agent stack"** publication, plus an in-person booth invitation. ## Referenced Sources & Third-Party Claims - **Cited systems/products (named, no URLs):** - OpenClaw / "open claw" — used as public case study; multiple referenced GitHub "issues" (state hole, overlapping writers/load-modify-save race, dangling tool call, approval drift, missing edge proof). - Hermes — cited as a personal agent example. - Codex, Cursor, OpenCode, Claude Code ("cloud code" in transcript) — cited as coding agent examples. - OpenAI Agents SDK — promotional reference. - **No external documents, papers, or datasets are cited.** - **Incident claims** reference unnamed "production incident" examples; provenance is via OpenClaw issues per the speaker's framing. ## Comment Availability - **Not specified** in the provided source material; no data on comment availability. ## Uncertainty Notes - Speaker name conflict: title says **Vinoth Govindarajan**; transcript self-intro says **Ben**. Not resolvable from source. - Transcribed speech contains apparent transcription artifacts (e.g., "I'm daughter of OpenClaw", "cloud code", "expose squashalty"); exact wording uncertain where flagged. Thank you for choosing to spend this session with me. My goal today is simple. I want to convince you all that most of the production failures are not most of most of the agent failures are not model failures. Those are harness failures. So let's start with one production incident. The user saw the reply. The system forgot it happened. This is a failure shape I want to start with. Not a hallucination, not a crash, not a bad answer. The user visible edge looked healthy while the durable record had a hole. In this example, the user asked the customer the agent to remember a refund for a customer. The assistant said it recorded the fact for the next turn. The interface looked normal. No red screen, no obvious failures. But the next turn cannot reconcept the fact. The user experienced success. The system inherited incomplete reality. Why this matters? The crash is annoying, but at at least it gives you a boundary. You know some you know something stopped. You usually see an error. You can often start from last known good point. Silent success gives you a lie. Delivery can succeed while the persistent fails. The user has no reason to doubt the reply. The operator has no obvious alarm. The next turn can still sound confident because the model is coherent but it is coherent over a broken history. That is why agent reliability matters. and agent relity cannot stop at model quality. Hi, I'm Ben. I work on core data and AI infrastructure at OpenAI. Before that, I worked on distributed systems at Apple and Uber. Outside of the work, I write the agent stack where I try to explain how the production AI and data systems work under the hood. I'm daughter of OpenClaw and this is not a openclaw product pitch. This is a pure system design talk. I'm using open claw as a public case study because its issues code docs makes the harness around the agent unusually visible. Here's a production contract for the talk. A model proposes the harness commits and the receipts proves it. The model can may suggest a message, a tool, a edit or an command. But model is not the production boundary. The harness owns the state transition, the authority check, the ordered commit and the receipt is the evidence that survives the term. Open clause is the case study and the contract is the takeaway. If you remember only three things from this talk, make it these. Own the state, order the mutation and prove the action. A fact needs only one owner and one replay path. Shared mutable state needs one ordered commit path and transcript is not the proof. A transcript tells you what the agent said. A receipt tells you what the system allowed, attempted, executed and what the user visible edge confirmed. To create a simple mental model, I created this car analogy of the harness. The model is the engine. It matters a lot. But nobody buys a production car by just looking at the horsepower alone. You also care about steering, brakes, road rules, dashboard, and a black box. The model gives you capability, but the harness gives you control. A powerful engine with no brakes is not autonomy. It is a liability with good acceleration. Here is the harness blueprint I wanted to uh discuss today. Every agent we know of like personal agents such as open claw or Hermes, coding agents such as codeex, cursor, open code or cloud code uses the same underlying architecture. Events enter from many surfaces a chat, web hook, timer or heartbeat or another external system. The control plane maps the events to a session key and the session key determines the state boundary. The session lane gives you one active writer for that mutable state. The runtime calls the models and tools. Tools act through approvals and policies and audit rail becomes the run receipt. This is the blueprint event session key throttle tools audit. The blueprint is a talk and the incident of the proof that each boundary matters. context is assembled uh in agent runtime does not usually remember in human sense. It is stateless. The harness rebuilds the working state for the each turn. The working set may include the transcript, session state, memory, policy, tool definitions. \[clears throat\] The model only sees what the harness supplies. If one input is missing or stale, the answer may still sound coherent. Coherence does not proves the working set was complete. These failures are not new. We already know about timeouts, retries, item potency, locks, ordering, and state ownership. What change is the agent setting? Now these failures sit around a probabilistic planner with dynamic plans. It's rebuilding the context for every turn. there are more event sources and it can act through more action surfaces. So these failures are familiar agents makes them easier to trigger and harder to explain. That's why agent harness matters. Let's talk about the first failure mode. This is the same failure mode I started this talk with. The user sees a success. The source of truth cannot replay it. Delivered is not remembered. In this stateole open claw issue, a telegram reply could succeed while the router turn was not written to the active context or transcript. The user saw the response. The log looked healthy. But the next turn had no durable record of the next change. A successful send proves transcript. It does not prove the future context. That distinction matters because the model can answer fluently over an incomplete record. The missing boundary was not intelligence. It was the state ownership. By owner, I do not mean a person. I mean the system of record whose persistent state becomes the truth. A calendar event belongs to the calendar system. A support status belong to the ticketing system. While a code change belongs to a workspace or repository on a conversation turn belongs to a session transcript and a user preference belongs to a memory store. Storage tells you where the bytes live. Ownership tells you who can reconstruct the reality. A replay is not a reliable memory until until a named owner can replay it. A system has to persist the turn. It has to name the owner or system of record and it has to make the replay possible. The question is simple. For every fact the agent might use later, who owns it and how would you replay it? If no owner can replay the fact, this the system did not reliably remember it. Once we know who owns the state, the next question is who's allowed to change it and in what order? Two correct rights can still produce one wrong outcome. and last writer wins is not a consistency model. In this overlapping writer open claw issue describes a load modify save race. Two callers loads the same old state. Each changes a different record. The sil the second save silently erases the first. The user may see a dismissed commitment return or receive a duplicate follow-up. Neither writer is malformed. Both operations are locally correct. The missing boundary is civilization around the commit. The invariant is not no concurrency. That would be too slow and it would miss the point. You can fan out sub agents. Parallel reads are fine. Independent retrieval is fine. Many sessions can also run at once. The rule is narrower and simple. One ordered commit path for one mutable state boundary. This mechanism may be a Q, a mutx, a transaction or a lock. You can use locks or mutx across the sessions and cues or transactions within a session. Be conservative of the commit time and not across the whole system. Users do not see cues or locks. They see behavior. A last correction feels forgetful. A stuck lane feels dead. And completion before delivery feels confused. Ordering is a product feature because users experience ordering books as personalities. Now let's talk about time in production. Silence cannot be neutral. Let's review. Uh the next uh failure mode is life cycle failure mode. The run waits for an event that cannot arrive. Silence is not a terminal state. In this dangling tool call issue, the session contains a tool call but no matching tool result. A process may have died. The connection may have dropped. A timeout might have happened before the results were recorded. The exact cause uh matters for debugging. The production failure is much simpler. The run is waiting for a event that will never arrive. New messages cue behind that silence. To the user, the agent simply looks stuck. Runs needs deadlines and cancellation. A deadline bounds the weight. Watchdog makes the stuck work visible. Tools needs time modes and error results. Channels needs recovery commands that do not wait behind the stuck work they are trying to fix. Every external boundary needs an ending. Success, failure, timeout, cancel, or max attempts. Most importantly, the receipt records the terminal outcome. So the next step does not have to guess. Bound the work before the work bounds you. Now let's uh we can move on from state to authority because a chat becomes risky when it becomes an action. Capability is not execution. The model can request an action. Requestability is not authority. Approval needs a shape. In this approval drift issue, expired approved call back was stated as retraable. The state call back state durable served restarts and blocked later channel work. The battle click existed. The valid authority did not. This is the mistake. Treating approval as a vague memory that the human was near the system or clicked yes. Approval as a scoped execution state, it must stay bound to the action it authorized. An expiration must terminate rather than loop. A useful approval object answers who approved in what session and run for which tool and for which arguments and for how long and with what outcome. It also point to the receipt. If those fields fall off during a retry replay or a channel call back the harness can no longer prove the action was being executed is the action being approved. The general lesson is simple. Capability is not execution. Least privileges narrows the tool surface. Scoped credentials ensures the right identity is used for the action. Approval and audit decides what happens before and after the execution. The model can reason about the boundary but it should not be the boundary. The model can request but the still the system decides. Finally, even if the tool says success, the user visible B may disagree. Internal component reports success. The user visible surface shows nothing. This is the inverse of the opening incident. We saw in this missing edging edge proof issue. The message tool reported success for a web chat or TUI run, but the message did not render. Normal assistant reply still appeared. The tool proved that the internal path accepted the request. It does not prove the user saw the result. That difference change the conversation. The agent may later say, "I already sent it." And the user may truthfully say, "I never saw it." Internal success is not external proof. Proof is a chain, not a claim. Model proposed something. Policy allowed or denied it. Execution attempted it. User visible edge confirmed or failed to confirm the outcome. The receipt preserves the chain. A transcript records what the agent said. The tool results records what one component claimed. A receipt records what the agent can verify at the boundary that matters. Let me recap all the incidents. Here are the file failure shapes you you to look for. A state hole, overlapping writers, dangling tool call, approval drift, and missing edge proof. For each one, let's ask the same question. What did the user see? Which boundary it broke? And what would the receipt have caught? Here is the audit I want you to run when you get back to your team. Pick one agent system, not all of them. One trace one production, one real production path and ask for the receipt. The audit has five questions. What woke it up? What state did it inherit? Which authority did it use? What executed? And what evidence survived? These questions expose squashalty. They turn a fluent conversation into an inspectable production run. First, what woke it up? A user message, web hook, timer, tool result, sub agent, or a replay. Name the trigger and its identity. Without that, you cannot reason about dduplication, order, or authorization. Second, which state did it inherit? transcript, session state, memory snapshot, policy version, and tool surface. The model only reasons over the working set the harness assembled. Third, which authority did it use? Record the actor, session, tool, run, arguments, scope, and lifetime. A model request is not permission. Authority should bind to a one pending action. Fourth, what executed? Record the tool or API call, arguments, attempt number, item potency key, and external results. This is a side effect boundary, not the poor summary of what the agent intended. Fifth, what evidence survived? Did the ticket get updated? Did the message got rendered? Did the file got changed? Did the calendar event exist? The receipt should end at the boundary the usual the user usually cares about. Now let's apply the opening incident. The same order to the opening incident. What woke it up a user message? What state it owned? That was the broken boundary. What executed the channel send? What evidence survived delivery? What did not survive the durable turn? Delivery survived while the state did not. That gap is the harness failure. The agent uh do did not need a better model. The model did not need a better prompt. The system needed a better harness with complete receipt. Let me recap the same three things I asked you to remember from the start of my talk. Own the state, order the mutation and prove the action. A better model helps inside the turn. Ownership, ordering, life cycle, authority and proof keep the system sane across turns. A model proposes the harness commits and the receipts receipt proves it. Once text can become an action, the useful question changes. Do not only ask whether the model can reason. Ask whether the system can own the state, order the mutation, bound the work, constraint authority, and preserve evidence. A loop can answer a turn and harness can serve up production. If you want to go deeper, scan the QR codes. The first points to the agent open AI agents SDK where all these um harness are already built in so that you can use to build your own agents and second points to the agent stack where I write about the production agents systems in more detail. I'll be at the open booth after this talk if you want to talk about your harness design. Thank you.