Talk by Hamza Tahir (co-founder and CTO of ZenML) arguing that checkpointing is an underrated primitive in AI agent infrastructure. Agents spend most of their time waiting while holding live compute; a "save button" freezes an agent to durable state, drops compute to zero, and restores it in milliseconds, resuming from the last checkpoint after crashes rather than re-running from scratch. Beyond cost savings, checkpoints enable reloading a finished run at any point in its trajectory, changing a variable (prompt, model, or tool), and re-running to compare outcomes. The talk also covers the supporting runtime infrastructure, such as sandboxes and Kubernetes-based systems, needed to run and improve large numbers of saved agents cheaply. # Video Identity Observations: "Your Agents Need a Save Button - Hamza Tahir, ZenML" ## Explicit Self-Description - **Speaker's role**: Hamza Tahir explicitly identifies himself as a co-founder of ZenML: > "ZenML is has been around for many years and is a player in the orchestration space. I'm one of the co-founders." - **Company/product claim**: The video presents **Kitaru** (transcribed variously as "Kitaru"/"Kitaro"/"Guitar Ru"), described as a new tool by the ZenML team: > "Kitaru is something we've launched recently, which allows you to have a runtime layer below your harness layer, and also connect to your traces." - **Open-source claim**: The tool is positioned as freely available: > "you can scan the repo. It's open source, free to use." ## Intended Audience - **AI/LLM engineers building production agents**. The talk assumes familiarity with agent harnesses, frameworks, observability spans ("Otel" spans), traces, evals, Docker images, sandboxes, MCP servers, and benchmarks like "towel bench" (likely Tau-bench). - **Practitioners concerned with production cost/quality trade-offs**, evidenced by discussion of model swaps, token costs, and escalation decisions. - **Conference audience**: The speaker references the venue directly: > "this conference is all about loops." ## Claimed Scope of the Channel/Content - **Topic scope**: Agent observability, checkpointing, and replay-based evaluation — framed as "checkpoint replay diff decide": > "it's basically evaluating using your production traces... evaluating using your production checkpoints." - **Positioning**: A thought-leadership/talk format arguing for a missing industry capability: > "this is what's missing today in the industry is that we don't have a clear connection between the observability spans... and the execution." ## Observable Presentation Patterns (Demonstrated, Not Claimed) - **Product demo format**: Walks through live use of Kitaru — replaying executions, changing a model to "GPT-5 Nano," mocking a "lookup policy" tool, and using a diff command with side-by-side comparisons. - **Cites third-party evidence**: References a **DoorDash blog post ("1st of June")** on simulated replay environments and a **BrainTrust study** on naive model swaps, plus a benchmark statistic ("a model that passes 60% of the time is only self-consistent about a quarter of the time"). - **Balanced framing**: Acknowledges limitations of the approach, e.g.: > "having a naive model swap usually or often times doesn't work." - **AI-assisted workflows**: Demonstrates using a "Kitaru MCP server" to have an agent analyze a cohort JSON report and flag issues. - **Actionable methodology**: Ends with a playbook — start from real runs, build cohorts, never ship from one or two replays, keep "a human in the loop." ## Relationship Between Claims and Demonstration - The **conceptual argument** (save button / checkpointing for agents) is supported by a **direct product demonstration**, making the video simultaneously educational content and a **product showcase for the speaker's own company's tool** — a pattern of content marketing integrated into a technical conference talk. # Delivery Observations — "Your Agents Need a Save Button" (Hamza Tahir, ZenML) ## 1. Opening with Direct, Rhetorical Questions The talk opens by immediately addressing the audience with a cluster of "what if" questions: > "Have you ever looked at your agent execution and asked yourself the question, why did it do that? What if it had done a different thing? Would it have been cheaper? Would it have been faster?" This question-driven framing recurs as a structural device — the speaker periodically restarts sections with new questions: > "And maybe at this point you might be wondering, but why even bother?" > "Okay, but this is just one point, right? What if I wanted to do this across a cohort?" > "What are the types of questions you might want to ask?" **Pattern (stable):** Question-then-answer structure is the dominant organizing habit of the talk. ## 2. Extended, Repeated Metaphor: The "Save Button" The central analogy — comparing agent checkpointing to document saving — is introduced early and deliberately recycled: > "your agents have a save button. We've had the save button for documents for decades now" > "Almost like an auto save, a command S, a control S in your agent." **Pattern (stable):** The metaphor frames the whole talk and is revisited near the end ("checkpoint replay diff decide"). ## 3. Frequent Filler and Self-Repair Speech The delivery shows heavy use of "uh," "um," and immediate self-corrections/restarts: > "Um and augments the traces that are emitted with actually the code execution" > "So, to do this in Kitaro, it's quite easy. All you have to do Now, to do this in Kitaro is quite easy." > "For for for the number of tokens consumed" > "So, it's it's it's trying to, you know, um, run an analysis" **Pattern (stable):** Self-restarts mid-sentence occur throughout, especially during the live-demo portion, suggesting looser scripting during demo narration. ## 4. Live-Demo Teaching Style with Hypothetical Framing During the demo, the speaker repeatedly frames changes as imagined scenarios before executing them: > "But imagine I wanted to do something different. Imagine I wanted to change the model" > "What if I wanted to mock a tool or or or if I wanted to change a tool call?" > "Now, what if I wanted to see them side by side, right?" He also acknowledges live-demo time constraints explicitly: > "this is going to take a while, so I'm not going to do it now" > "While this is going on, um, maybe we can go back and continue our presentation." **Pattern (stable within demo section):** Consistent hypothetical-then-execute cadence. ## 5. Personal Anecdote and Honest Caveats The speaker distinguishes personal observation from product claims and volunteers counter-evidence: > "to be very honest is that while we've been doing it this with our users and customers, what I've personally seen is that having a naive model swap usually or often times doesn't work." **Isolated but notable moments:** - "what I love to do personally, is use the Kitaru MCP server" — first-person preference framing. - "So, the verdict is don't ship" — a concrete, negative result shown from his own demo, framed as honest reporting. ## 6. Mnemonic Summarization Near the end, the methodology is compressed into a repeated formula: > "So, checkpoint replay diff decide." > "you just ship, route, and hold" **Pattern (limited repetition):** Used in the closing section; not frequent enough to confirm as a general habit beyond this talk. ## 7. Vocabulary and Recurring Terms - **Technical terms used repeatedly:** *checkpoint/checkpointing*, *replay*, *cohort*, *trace*, *state*, *what if*, *diff*. - **Product name:** "Kitaru" (spoken variously as "Kitaro"/"Kitaro"/"Guitar Ru" — the transcript shows inconsistent renderings, e.g., "a tool called Kitaru" and "If you want to use Guitar Ru"; the variation may reflect pronunciation or transcription, uncertain). - **Audience-aware phrasing:** references the venue context: "Uh this conference is all about loops. So, this is nothing different." ## 8. Uncertainty Notes - The "closing the loop" and "loops" framing appears tied to this specific conference context; cannot confirm it as a stable habit. - The DoorDash citation is delivered as evidence but with secondhand framing ("has a a blog post on the 1st of June"); the delivery habit is citing external studies (DoorDash, BrainTrust, "towel bench") to support claims — this recurs at least twice, so it appears to be a genuine pattern. # Observation Document: "Your Agents Need a Save Button" — Hamza Tahir, ZenML ## Speaker - **Hamza Tahir**, co-founder of ZenML, presenting on agent execution checkpointing and replay. Tool demonstrated: **Kitaru** (transcribed variously as "Kitaru"/"Kitaro"/"Guitar Ru" — same tool), launched by ZenML. ## Core Thesis - Agents lack a **persistent save state** analogous to "Control+S / Command+S" that documents have had since the 1980s. - The closest existing artifact is a **trace**, but Tahir argues traces are: - **Disconnected from the runtime** — variables in state, in-flight file systems, code decisions, and the code itself are lost. - Reduced to a **read-only stamp** in a separate tool, far from the actual code. - Missing a clear connection between **OpenTelemetry observability spans** and execution. ## Why It Matters: Replay and "What If" Questions - A save capability enables **replay**: going back in history to ask what-if questions, e.g.: - **Swapping the model** (e.g., to a cheaper open-source model). - **Mocking or overriding a tool's return value**. - **Intentionally degrading** a component to test failure behavior. ## Emerging Stack - A category is emerging that sits **on top of agent harnesses/frameworks** with a **durable runtime below**, augmenting traces with code execution and surrounding context to complete system state. - Production systems already contain what's needed: traces plus state checkpoints from the runtime. ## Proposed Methodology: "Checkpoint → Replay → Diff → Decide" - Applies to a **cohort of runs** (e.g., expensive, slow, or risky ones), not single runs: 1. Replay a change across the cohort. 2. **Diff** against a known baseline. 3. Decide, route, and ship — "closing the loop on your evals," i.e., **evaluating using production checkpoints**. - Tahir states he has seen this methodology scale, citing a **DoorDash blog post (June 1st)** describing a simulated environment replaying customer bots: - Reduced analysis from **hours to 5 minutes** with hundreds of simulations. - **90% fewer hallucinations**; simulations within **two points** of production behavior, per Tahir's account. ## Demo Positions (Kitaru) - Checkpoints capture **configuration, code, input/output artifacts, and environment** (Docker image or sandbox). - **Model-swap replay**: changing a checkpoint to GPT-5 Nano; prior checkpoints are skipped since state is already held, execution resumes from the fork point. - **Tool-mock replay**: replacing the `lookup_policy` tool with a function returning a different policy; model held constant. Tahir argues having the code makes tool mocking easier than when disconnected from the codebase. - **Diff view**: original vs. forks; baseline identical, diverging behavior visible afterward; artifacts comparable side by side. - Outcome: first two runs → "needs review"; third (changed policy) → "safe to answer." Tahir qualifies: "this might or might not be good in your scenario." - **Cohort replay** with JSON output; analysis via the **Kitaru MCP server** using an LLM agent to analyze cohort data — Tahir argues this matters because analyzing thousands of runs manually is infeasible. ## Qualifications, Caveats, and Contradictions (preserved as stated) - **Naive model swaps often don't work**: "having a naive model swap usually or oftentimes doesn't work." Cost savings can be a **false economy** if the support agent stops resolving requests — citing a **BrainTrust study** on value created vs. spend. - **Single replays are anecdotes**: citing the **towel bench**, Tahir states a model that passes 60% of the time is only self-consistent about a quarter of the time — therefore cohort analysis is "way way way better" than one estimate. - Replay at scale **"can get very expensive"**; one must be smart about what to replay. - **Demo verdict**: although the single replay looked cheaper with the same result, the cohort analysis concluded **"don't ship"** the cheaper model for this data — explicitly qualified with "this might be different for your data." ## Recommended Playbook (as stated by Tahir) - Start from **real production runs**, not synthetic ones. - Build **cohorts that matter** (expensive, long, risky). - **Never ship based on one or two replays.** - Replay at scale; ship, route, hold; automate the loop (ideally with an agent) while keeping a **human in the loop**. ## Closing - The capability requires modeling agents in a **runtime that checkpoints state** and can **replay from code** with different scenarios. - Kitaru is **open source, free to use**; Tahir invites feedback. # Speaker Attribution Record ## Video: "Your Agents Need a Save Button - Hamza Tahir, ZenML" ## Speakers - **Hamza Tahir** — **Creator / presenter / demonstrator** - Sole speaker throughout the transcript. - Self-identified as **co-founder of ZenML**: > "I'm one of the co-founders." - Presents a conference talk (he references "this conference") including a live product demo. ## Roles & Provenance Facts ### Speaker Affiliations (self-declared) - **ZenML** — company; speaker is a co-founder; described as > "has been around for many years and is a player in the orchestration space." - **Kitaru** (transcript also rendered "Kitaro" / "Guitar Ru" — spelling varies in transcript) — new tool launched by the ZenML team, described as open source and free to use. ### Referenced Third-Party Sources & Claims - **DoorDash blog post** (dated June 1 by the speaker): claims of replayed customer bots in a simulated environment; reductions from hours to ~5 minutes with hundreds of simulations, 90% fewer hallucinations, within two points of production. *(Claim attributed to DoorDash, not verified here.)* - **BrainTrust study**: speaker describes it as showing potential "false economy" of naive model swaps. *(Third-party claim, not verified.)* - **"towel bench"** (likely Tau-bench; named as spoken): cited statistic that a model passing 60% of the time is self-consistent about a quarter of the time. *(Third-party claim, not verified.)* - Mentions of **Otel/Odel observability spans**, **GPT-5 Nano**, **MCP server** — referenced technologies, not sources of claims. ## Promotional Segments (Flagged) - **Kitaru demo section** — extended live product demonstration of the speaker's own company's newly launched tool (checkpointing, replay, diff UI, replay cohort command, MCP server analysis). - **Closing call-to-action**: > "If you want to use Guitar Ru... you can scan the repo. It's open source, free to use, and we'd appreciate the feedback and love." ## Availability & Limits - **No curated metadata supplied**; only the transcript was provided. - **Comment availability**: not indicated in source. - Exact URLs, slides, repo links, and timestamps are **not present** in the source. - DoorDash and BrainTrust references lack titles/links in the transcript — cited by name and date only. Have you ever looked at your agent execution and asked yourself the question, why did it do that? What if it had done a different thing? Would it have been cheaper? Would it have been faster? Well, you can do all these things if your agents have a save button. We've had the save button for documents for decades now. Since the 1980s, people have been used to pressing control S, command S uh or auto saving while you're working to have a persistent state. But agents, they don't have that today. The only thing we have which is closest is a trace. A trace gives you the emitted telemetry data of how an agent calls tools in the input and output of that state. Now, while this is a good start, it is actually very disconnected from the runtime in which these agents actually execute. So, all the variables that are in state, all the file system that is in in flight, um the decisions that it makes in the code, uh the actual code itself, all of that is lost and it is only stamped as a read-only trace by the end, which is sitting in another tool far away from where the actual code is. And I think this is what's missing today in the industry is that we don't have a clear connection between the observability spans that are emitted with Odel and the execution. And maybe at this point you might be wondering, but why even bother? Why do I need to have a save button? Well, save allows you to replay. You can go back in history and ask the what if question. What are the types of questions you might want to ask? Well, you might want to swap the model. Maybe you use a an open-source model that is cheaper. Maybe you mock a tool and you override the what it returns. Maybe you degrade it intentionally to see what would happen if things are wrong. And these sorts of questions are only possible if you have that state. And there is a category of the stack emerging which actually allows that. And this sit on top of the harness sit on top of the frameworks that allow you to create agents. And put a durable runtime below that. Um and augments the traces that are emitted with actually the code execution and the things that are around it to actually complete the state of this of the of the system. And the good news is that once you have such a system in production, you already have the information you need to ask those questions that are relevant to making your system better, cheaper, and faster. Production already has the traces. It already has these state checkpoints ideally from the runtime that can allow you to go back in time and ask those questions. For example, let's take an agent example which does a customer resolution and refunds after a chargeback dispute. Well, you can then see if the order status is changing languages or whether the request should have been escalated or maybe a smaller model would have handled it if the runtime is checkpointing each of the state as it goes along. Almost like an auto save, a command S, a control S in your agent. And once you have that, you can even close the loop. Uh this conference is all about loops. So, this is nothing different. You have a cohort of runs that you think maybe matter because maybe they're too expensive, they took too long. You replay a change, you diff it, you see what would have happened when you have the baseline, which you know what happened in the first place, and then you decide and you route and you ship back. That's closing the loop on your e-vals. It's It's It's basically evaluating using your production traces. So, it's basically evaluating using your production checkpoints. So, checkpoint replay diff decide. And this is really the methodology that that I've seen and I've seen others do, uh which has really scaled. For example, DoorDash. DoorDash has a a blog post on the 1st of June where they talk about having a simulated environment where they replayed customer bots. And they've done what-if scenarios and seeing how they could have made it better. And where And where it used to take them hours and hours to to do this, now they've reduced it to 5 minutes uh with hundreds of simulations, have 90% less hallucinations, and they're still two points within what they've seen in production. So, the simulations are pretty good because they're grounded in what's already happened. So, we're going to just walk through this um in an example, and we're going to see how this works. Uh for this demo, we're going to be using a tool called Kitaru. Kitaru is a very new tool that is launched by the team at uh ZenML. ZenML is has been around for many years and is a player in the orchestration space. I'm one of the co-founders. And Kitaru is something we've launched recently, which allows you to have a runtime layer below your harness layer, and also connect to your traces. Um and do all the checkpointing that we were just talking about and then running replay scenarios. So, you can see here I have all of my I have a support uh agent which looks at my customer requests and escalates it when it needs to to humans. You can see various things as you might expect. Um, every tool call uh can see a timeline view if I wanted. And here the difference is that if I if I click on a particular um um checkpoint like a tool call, I can see the configuration, where it ran, uh the code it it took um and the artifacts which came in and out of it. Um and this combination of code and the artifacts that it created and the environment in which it ran in, whether it was a Docker image or a sandbox, those are all snapshotted in state here between the checkpoints. And you can see that here in this particular example, there was a few tool calls every time it went to the LLM, and I can actually see um you know, how long it took. But imagine I wanted to do something different. Imagine I wanted to change the model if I maybe at this point I wanted to use a cheaper model. Would it have done the same stuff afterwards? So, to do this in Kitaro, it's quite easy. All you have to do Now, to do this in Kitaro is quite easy. All you have to do is you have to take your execution and replay it at a particular point. So, I'm just going to copy this over and I'm going to put it in my terminal and see what happens. So, here what I'm doing is essentially I am saying, "Okay, after this particular tool call, change the model to GPT-5 Nano, which is obviously a bit cheaper." Um and then what's going to happen here is there's going to be a new execution that's launched from 71 and you can see the first three checkpoints are skipped. So, they're all skipped because Kitaro already has the state of all the checkpoints before that. All it needs to do is just change this particular checkpoint and then execute start executing from here. And this is really cool because now I can see what would have happened in this scenario if there was a cheaper model. So this looks pretty similar to me, but what if I had wanted to do a even different change? What if I wanted to mock a tool or or or if I wanted to change a tool call? Well, in order to do that, you can mock up for example the lookup policy tool. And here it's also very easy. I can just go back here. Replay. And this time, rather than changing the model, I am changing the lookup policy. And I'm mocking it with another function in my code base, which returns a different lookup policy. And I'm just trying to understand if the policy had changed, what would have happened. This time I'm holding the model constant. Now, the interesting thing here is that because I have the code, it's very easy for me to do tool calls and to change these particular things and do more experiments than I would have had if I was completely disconnected from the code base. And here you can see that the code base is a little bit different, so I see my logs here. They look a little bit different. And here you can see that I got a slightly different artifact maybe from the tool call, and it published the thing. So now again, I have three runs. I have the original run and I have the two replays. Now, what if I wanted to see them side by side, right? So Kitaro actually has this very handy diff command that lets me give an original ID of an execution and then allows me on the other side to actually see them side by side. So I can see what happened. So there's a few warnings here about some artifacts, but in a second it will go and give me a URL. I can copy this URL and I can actually put it directly here. And now I have a very nice um comparison of the original and the two forks. Um I can see a bunch of things here, but I think what's really interesting is this view. So, in this view, you can see that the first part of the the baseline is the same, right? So, these things were skipped. They were exactly as it is. The state is exactly where it was. Um but now, right after that, in the third replay, it's a little bit different, right? The tool call happened a bit differently because we used a different policy, and then something changed after here. Um here, it took a little bit longer. And you can really start looking at the final result. And if I click on the final result, I can actually see the artifacts side by side. I can see what decision um it actually ended up making. So, you can see that it uh here, it restricted the count charge. Here, it also restricted it. Here, it also restricted it. In the first two, it actually needs review. In the third, because we changed the policy, it's safe to answer. So, this might or might not be good in your scenario, but if is this is what you expected to see? Well, maybe. Because the these two were cheaper at the end of the day. For for for the number of tokens consumed, uh they were cheaper. Input and output. And you can see a bunch of detail here uh in the in the UI, which might be useful for these for these analyses. Okay, but this is just one point, right? What if I wanted to do this across a cohort? What if I wanted to have a bunch of runs, right? Which actually um maybe sorted by cost. So, maybe I took all of my expensive ones. Then I wanted to do one change across the entire cohort. So, change the change all the tool calls that happened uh that happened for this particular set of configurations across the cohort. Or change the model itself um and use a cheaper model across the cohort. So, this gives me a bigger distribution of information and replays that I can use. Um and now, I can just this. Um, and the way you would replay it is you can use the replay mini command where you actually um, start from a particular point and you do the same as you did before just across the cohort. Um, this is going to take a while, so I'm not going to do it now. But once you execute it, in this particular case, I just emitted it to JSON, so I have a very nice, uh, JSON here, and this JSON gives me a bunch of things. So, this is a bit hard to do the UI, right? So, there's a lot of things going on. You can't just do the many, many, many comparisons here. Um, but what you could do, and what I love to do personally, is use the Kitaru MCP server. And here what you do is you can just say, "Hey, read this you read this JSON report and do an analysis on what you think I should be doing across the cohort." And I think this is what What is really important is to be using agents and LLMs to analyze these cohorts across a plethora of data because at some point uh, I mean, 10 is probably easy to do, but what if you have thousands? Um, and doing thousands and thousands is hard, and this is where skills and MCP servers get really relevant and having the runtime be queryable and go into your execution and fetch the artifacts is very important. So, it's going to be it's going to be uh, doing a lot of things. So, it's it's it's trying to, you know, um, run an analysis around these decisions and it's going to flag any um, red flags that that happen. So. While this is going on, um, maybe we can go back and continue our presentation. So, um, Here you can see, yeah. Yes, if you change the model, it can get very cheap. And of course, you don't want to do it across just one sample, but across many because then you can see a bigger variation. I think one thing which is important here is and to be very honest is that while we've been doing it this with our users and customers, what I've personally seen is that having a naive model swap usually or often times doesn't work. So just changing for example to a cheaper model and just looking at the cost one single dimensionally, obviously it could be that you're you're spending a lot less money, but what happens if your support board is not resolving the requests, right? So this is a study from BrainTrust. Excellent study where they actually looked at that and they saw that there could be a false economy if you do a naive model swap because it might look on paper that you're faster and you're cheaper, but at the end of the day you have to look at the value created, right? So it's a trade-off between how much money you want to spend and and yeah, the result. And also, if you look at the towel bench, what we must understand is that a model that passes 60% of the time is only self-consistent about a quarter of the time. So which basically means that one replay is just an anecdote and having a cohort analysis is way way way better. Um and because then you can really see across a population and across scale what would have happened not just looking at one estimate. This can get very expensive of course and this is where you have to be really smart about what you replay and have tooling that really helps you. And this you can really bake into your production process as well, right? So again, if you if you take a step back and look at the playbook, you can start from real runs, not synthetic, but real runs, real production uh state. Build cohorts that matter. Um maybe take the expensive ones, maybe take the long ones, maybe take the risky ones. Um never ship anything by just replaying one or two things. Um and just do this at scale and uh ship, route, and hold, and try to automate that loop as much as possible. Maybe there's an agent that's doing that for you. Um that's even better. But you just have a human in the loop at the end. But you just have a human in the loop at the end. Now, let's see where our cohort analysis has gotten us. Okay, so we're done. So, the verdict is don't ship. So, even though it looked like from a single replay that it was cheaper to do and we reached the same result, across a bunch of those support cases, you actually saw that our agent concludes that you shouldn't be using a cheaper model in this particular case for your data. But this might be different for your data. In conclusion, um if you want to be replaying your agent executions and answering the questions what if I had done something different while designing this agent or what if the agent could be driven to do something different? Well, you can do this if you model your agent with your harness in a runtime that can checkpoint state and is able to replay that state from code with different scenarios. If you want to use Guitar Ru, the the the tool that I showed that allows you to do it, um you can scan the repo. It's open source, free to use, and uh we'd appreciate the feedback and love. Thank you so much, and see you guys on the next one.