Nick Nisi and Zack Proser (WorkOS) demonstrate how to build verification loops that let coding agents check their own work, covering checkable goals, CLI health checks, browser snapshots, LLM judges, and eval baselines so failures drive the next fix. Includes a live demo app with a follow-along repo. Chapters: 00:00 Why 'done' isn't done; 03:17 What it takes to trust agent output; 09:34 Give agents goals they can check; 17:31 Closing the feedback loop; 28:41 Making everything feedback-loopable; 36:29 The moment the agent knows it's not done; 39:33 LLM judges and eval baselines; 53:32 Q&A. # Video Identity Observations: "Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS" ## Channel Presentation Context Observed ### Format & Setting - This is a **live workshop presentation** at a conference ("Hey, welcome. Welcome to Beyond Done... this workshop"), delivered in New York with an in-person audience. - Presenters self-identify: **Nick Nisi** — "I'm an engineering manager at WorkOS... I live in Omaha, Nebraska"; **Zach Proser** — "I'm an engineer on the Applied AI team at WorkOS." - The channel/personas present as **WorkOS practitioners**, not product marketers: "This is something that we build and we hold our own tools too as well." ## Intended Audience ### Addressed Audience - **Professional developers already building with AI agents.** Evidence: - "Show of hands, who in this room is building with a ton with agents... more than half of the room." - "How many folks are running multiple terminal tabs or multiple chat sessions or multiple agents, like everybody that put their hands up." - The content assumes fluency with PRs, CI, evals, MCP, Playwright, and LLM terminology — no basics are explained. ### Audience Problems Reflected - The channel frames its audience's pain as **managing many parallel agent workstreams**: "these agents that we're now all managing as like herds of insane token machines." - References shared professional anxieties: "I don't want to be babysitting the PR." ## Claimed Scope ### Core Thesis Claimed - The workshop claims agent **verification is foundational to scale**: "verification now... is one of the key foundational things to focus on." - It explicitly positions itself as a **workshop on trusting agent self-reporting**, motivated by an incident: "an agent just started deleting production databases. And in 9 seconds, everything was gone... that is what we want to help you prevent here today." ### Scope Boundaries the Creators State - No universal tool is offered — **application-specific verification is the claim**: "we couldn't give you a tool at this time that works generically for every single application." - Strong emphasis on **experimentation over prescription**: "It's too early to really classify on anything, so experimentation is key." - Goals are stated as a **gradient of trust**, not full autonomy: "it's not just going from zero to one, but it's a gradient." ## Self-Description Evidence ### How the Presenters Describe Their Own Practice - Radical agent reliance, stated hyperbolically: "I haven't written code since July 30th, 2025... I saw PHP and I was like, 'Never again.'" - Parallel agent workflows as normal: "Normally, it's like 10 to 15 [agents running]." - Claimed team results: "We have people on our team that are 900 PRs a month and no incidents and improving KPIs." - Personal tool experiments are showcased: Zach's **CTX** ("a simple web app that runs on Cloudflare. It was vibe coded initially") and **Riker** ("its job is to babysit it"); Nick's **outcome-driven development**. - Reliance on third-party review tooling: "We use Reptile. Reptile gives me an X out of five score... We've gotten to that level of trust with Reptile." ### Stated Worldview / Forward-Looking Claims - The channel presents a thesis of **human role transformation**: "we're all becoming agent harness managers... I'm almost gardening multiple code bases and letting them self evolve." - Predicts **RSI/self-evolving codebases** as the future: "we're starting to see more and more harnesses come online that have the concept of RSI or recursive self-improvement baked into them." - Claims early-mover advantage from data preservation: "there will be a specific advantage for folks that have preserved clean data sets." ## Demonstrated (Not Just Claimed) Patterns - **Interactive teaching method**: a live repo workbench with exercises, glossary, QR code, and an "ask anything" AI trained on the slides — plus an audience-interactive "saboteur and detective" bug-hunt game. - **Live demos with honest failures**: demos break on stage ("it failed. I didn't actually run the test... if we had run the fix, it would show that"), and the presenter jokes, "I need some verification tests around my... verification examples" — reinforcing the channel's verification-first ethos through demonstration. - **Audience Q&A engagement**: multiple live questions answered on CI enforcement, hooks, agents.md, and production rollout monitoring, indicating a community-oriented, practitioner-to-practitioner presentation style. ## Summary This video presents the channel as **WorkOS engineering practitioners teaching advanced agentic-development practice to working developers**, with claimed scope centered on a single thesis — *agents must prove their work with verifiable evidence* — delivered through live, hands-on, failure-inclusive workshop demonstrations rather than product pitches. # Delivery Observation — "Teach Your Agent to Prove It's Done" (Nick Nisi & Zack Proser) ## Format & Structure - **Dual-presenter workshop format.** The two speakers alternate frequently, handing off mid-topic with short affirmations: > "Yep. And at that point they're able to pass on their own…" > "Yeah, indeed." - **Time-aware framing.** They narrate pacing aloud: *"We got a lot to pack in"*, *"We got less than a minute"*, *"we've got just over 5 minutes. I'm going to pause and see if there are additional questions."* - **Live demo habit.** Nearly every concept is followed by running a real command: `pnpm snack up`, `p snack doctor`, `p snack snapshot`, `p snack judge`. Demos occasionally fumble, and they name it honestly: > "I can't drive Chrome, apparently." > "It's funny, I need some verification tests around my verification my contrived verification examples." ## Recurring Vocabulary & Phrases - **"Verification" / "verify"** is the dominant lexical anchor, appearing throughout: *"verification loop"*, *"centralized authority for verification"*, *"custom verifier CLI"*. - **"Receipts" / "show me the receipts"** recurs as a metaphor for concrete evidence: > "Status comes from records, not model pros… it's, you know, show me the receipts." - **"Foundational"** is a repeated intensifier: *"foundational eval"*, *"this being like a foundational concept now"*, *"foundational to achieving true scale"*. - **"Show of hands"** opens audience-polling moments at least twice: *"show of hands, who in this room is is building with a ton with agents"*; *"how many of you write good error messages? Show of hands."* - **"Anecdotally"** prefaces experience-based claims: *"anecdotally, this is what we're all experiencing at WorkOS"*; *"just anecdotally it is um something that we all kind of have to take a step back…"* ## Extended Metaphors (sustained, reused) - **Gardening / self-evolving codebases:** *"I'm almost gardening multiple code bases and letting them self evolve"*; later *"the gardeners making sure that they're evolving correctly."* - **Agent embodiment:** *"These tools give your agents their eyes… the tools… gives it the body. The LLM is the brain."* - **"Lanes of work":** *"the 18th… lane of work that you're driving today"*; *"all of the different lanes that I started that day."* ## Teaching Habits - **Contrast pairs (old way vs. new way).** Nearly every section contrasts human-era habits with agent-era practice: *"refresh the page, is the application still visible… and then what is the delta between doing this and… saying 'Here's three paragraphs of text. Go execute these instructions.'"* - **Escalating specificity demos.** When explaining prompts vs. scripts, they walk a ladder: *"always verify the change in a browser… 'Remember to reload and check'… 'Check that the data persists'"* — each labeled by increasing specificity. - **Concrete numbers for credibility.** *"900 PR's a month and and no incidents"*, *"10 to 15"* agents normally, *"3 or 4 days"* agent sessions, *"18 or 30 terminal tabs."* - **Audience participation.** Direct Q&A mid-session (*"Can anyone tell me what the bug is? Any ideas?"*), plus collaborative brainstorming: *"maybe in the before times before an LLM"* — collecting audience answers ("refresh the page," "network") and affirming them: *"Absolutely, yep."* - **Playful interjections with the co-presenter.** *"I'll put in zach@spamhim.com." / "Thanks, buddy."*; *"Sorry, we don't type anymore."* ## Tone - **Conversational, hedge-heavy, self-deprecating.** Frequent fillers (*"like," "uh," "you know," "kind of"*) and jokes at their own expense: *"I saw PHP and I was like, 'Never again.'"* - **Enthusiastic forward-looking register.** *"It's a really exciting time"*, *"They're all really good"*, *"it's going to just get better."* - **Memorable coined framings** used to make ideas stick: *"herds of insane token machines"*, *"hand chiseling code"*, *"mount and climb that evidence ladder."* ## Stability Assessment - **Stable patterns** (well-supported by repetition): verification-centric vocabulary, receipts metaphor, live-demo-then-explain rhythm, contrast-pair teaching, audience polls, hedged conversational tone. - **Isolated moments** (single occurrences, not clearly habits): the *System 1 / System 2* psychology analogy and the yellow-light example; the *saboteur/detective* game; the colored-light and Airbnb jokes. These appear once and shouldn't be treated as signature delivery features without more material. ## Uncertainty Notes - Speaker attribution in the transcript is ambiguous (alternating `>>` markers without names), so individual speaker-specific habits (e.g., who says "anecdotally") cannot be reliably assigned. - Some numbers (*"900 PR's a month"*, *"thousands of PRs ostensibly a month"*) are anecdotal claims from the speakers; they are quoted as delivery content, not verified facts. # Observation Record: "Teach Your Agent to Prove It's Done" — Nick Nisi & Zack Proser, WorkOS ## Speaker Identification Two speakers are identifiable by explicit self-introductions in the transcript; remaining turns alternate between them without always being individually labeled, so unattributed turns are recorded as "Speaker (unlabeled turn)". - **Nick Nisi** — Engineering manager at WorkOS, Omaha, NE; states he has "not written code since July 30th, 2025." - **Zack Proser** (transcript spelling varies: "Prosser") — Engineer on the Applied AI team at WorkOS. ## Core Thesis > "If an agent can verify its own work... if an agent can't verify its own work, nothing else matters." - Attributed to **Lauren Tan** (described as "a famous potato on X"); quoted by a speaker (unlabeled turn). - Both presenters converge on the position that **verification, reliability, and evals** are foundational to scaling agentic development, especially as "harnesses... have the concept of RSI or recursive self-improvement baked into them." ## Substantive Claims and Reasoning ### 1. Human code review no longer scales - **Speaker (unlabeled turn):** Reviewing agent PRs like a normal code reviewer "takes a long time," creates a poor signal-to-noise ratio, and abstracts humans from code they no longer write. - **Zack:** With "18 lanes of work" and "diffracted attention," subtle bugs creep in — agents report "the code looks good, it builds, it runs, the tests pass, and it's still wrong." He characterizes agents as "herds of insane token machines." ### 2. UI-only verification is insufficient (demo) - **Nick (demonstration):** The "Midnight Snack Club" app showed a success message after email signup, but "if you reload that page, nothing's there" — the update "didn't persist... We didn't check the other half." - Stated lesson: checking the UI alone is not verification of done. ### 3. Verification is a gradient, not binary - **Nick:** "It's not just going from zero to one, but it's a gradient... getting us closer to one... is the goal." Until solved, "we're not going to trust these agents to really accelerate our work to its fullest extent." ### 4. Status comes from records, not model prose - **Nick:** "Status comes from records, not model pros [probably 'prompts']" — "show me the receipts... actual screenshots, videos, test results, smoke tests... to prove to me without me having to look at the code." - He adds he cares about the **result**, not the code: "It can have bad code, honestly, and deliver that result pretty well." ### 5. Task decomposition over giant PRs - **Zack (framed as anecdotal):** "What is working, you know, essentially perfectly for folks is decomposing tasks really cleanly," giving each agent exact done-criteria. Claims team members ship "900 PRs a month and no incidents and... improving KPIs." Explicitly framed as anecdotal ("anecdotally," "I'm not approaching this code base anymore in the traditional way. I'm almost gardening multiple code bases"). ### 6. Zack's tools: CTX and Ryker - **CTX ("context")** — a web app on Cloudflare, "vibe coded initially. It works." Evolved from a cross-agent memory layer → secrets handout → centralized verification authority. Agents return via MCP saying "am I done? Please verify it"; CTX runs evaluators (e.g., checks the GitHub API that a PR "truly does exist and is healthy and... all the merge checks pass"), marks workflow nodes satisfied or redirects the agent with failure details. - **Ryker** — an "engineering-focused Grokbot or Muse" that "babysits" PRs: builds checks, runs tools, keeps evidence, decides when to stop, and alerts Nick only when needed — keeping him "out of... the minutia of shipping the final 10%." - Both explicitly framed as **experiments**: "It's too early to really classify on anything, so experimentation is key" (Nick). ### 7. Prompts alone are too vague; scripts are deterministic - **Nick:** Vague prompts ("always verify the change in a browser") leave "a lot of interpretation." A deterministic script that feeds output back "closes that loop" so the agent can "run it again, check, verify... until there's no more errors." - **Zack:** LLMs are non-deterministic ("even giving 30 agents the exact same prompt... potential for variance"); pinning to "a single command... backed by actual bash calls" is "significantly more deterministic." Claims high-output practitioners like Lauren Tan "have mentioned that this is the first thing that they concentrated on that actually unlocked the velocity" (hearsay, flagged as such). - **Nick:** If a web app lacks "a bespoke CLI that allows any agent to fully flex the functionality," you'll have "a much rougher time." ### 8. Verifiers may matter more than the application itself - **Nick:** "It might even be more important than your software... the way that you verify it... If you have all of those verifiers in place, then your source code doesn't matter as much." Cites the Bun team's Rust conversion and "Theo just last week converted it to Rust again" as informal evidence. ### 9. The "snack" workshop tool - **Nick:** A non-reusable template with commands: `up`/`down` (health), `doctor` (environment checks), `drive` (join, reload, reset, snapshot, storage, console, screenshot, evals), `prove` (deterministic checks), and `judge` (system-one model judgment). - **Caution (Nick):** "A successful screenshot isn't proof... it might just be like another lie" — screenshots may not reflect the actual path taken through the UI. ### 10. Strict text-matching breaks; system-one judges fill the gap - **Nick (demo):** A fix that persists data can still "lie" — the confirmation says "you're in, your seat is confirmed" when the expected text was "waitlisted," failing strict one-to-one matching. The proposed answer: "those system one models." - **System 1/System 2 explained (unlabeled/Zack):** drawn from psychology — fast autonomous decisions vs. deliberative thinking; applied to fast, cheap judgment models vs. reasoning models. Mentions **Jev** and **Cleft from Cloudflare** as recent fast models. ### 11. Don't wrap determinism around LLMs — constrain, don't prescribe - **Zack:** "We're trying to wrap determinism around LLMs, but we actually don't want to do that... constraints can equal freedom... It's not about being overly prescriptive... that can actually lead to lower performance in some cases." - **Nick:** Frontier models (named as "Fable, Astra, Saul 6.1, Opus 5.5") are "really good at being... creative"; give the goal, not the path, and "they might wander and... catch more edge cases." Notes this appreciation is recent ("we've only come around to really appreciating [this] in the latest batch of Frontier models"). - **Zack** references Nick's experimentation with "outcome-driven development." ### 12. Good error messages enable self-healing - **Nick:** Agents "can interpret and run with that error really well"; failures should be "explicit about why it failed" with "as much detail as possible." ### 13. Keeping receipts as future training/eval data - **Zack:** Every agent run is "very valuable data right now" (feedable to LLMs for fixes) and in the future for RSI loops: "you've got a data set of exactly how your system performed that you can create very comprehensive evals for and have agents hill climb on." - **Competitive claim (Zack):** "There will be a specific advantage for folks that have preserved clean data sets... If you're not sitting on a mountain of that and your competitors are... they're going to be able to go significantly faster." ### 14. Evidence ladder as living spec - **Nick:** Unit tests, evals, smoke tests, plus model-interpretable checks enable regression-free climbing: "mount and climb that evidence ladder... That really becomes the spec that shows what our app does." ## Q&A Positions (audience + presenters) - **On models writing JSON for determinism:** Yes — one presenter uses unit tests around model output "to make sure that it doesn't drift over time." - **On skills vs. enforcement (audience member's stated position):** They "would personally almost lean away from skills" because model compliance is optional; prefer CI enforcement. - **Presenter response:** Checks can run in CI, as hooks (git hooks "probably the last one I would reach for"), or via a `pi` extension; easiest team-wide approach: "put an agents.md, not a cloud.md, in your... app" with verify-on-change instructions. - **Local vs. CI:** Local gives "faster feedback loop," enabling parallelism — one presenter reports "normally... 10 to 15" agents running. Goal: agents prove done to themselves so he isn't "re-prompting." - **CodeRabbit reliance (unlabeled presenter):** "Reptile [CodeRabbit] gives me a five out of five and all of my CI is passing, I feel extremely confident" — but also notes the failure mode: "I could add 10,000 lines of absolutely nothing... and it goes into production... Now it's just an extra 10,000 lines of nothing." Hence verification keeps agents on track. - **Preview/production monitoring (Zack):** An emerging pattern of "at least one agent specifically concerned with that production rollout... watching DataDog, watching feature flags... this is actually bad and we're reverting it." He acknowledges "queasiness" but says agentic production rollout "seems inevitable." ## Contradictions, Qualifications, and Tensions (preserved) - **Anecdote vs. evidence:** Multiple high-scale claims (900 PRs/month, thousands of PRs/month by Lauren Tan and others) are explicitly flagged as anecdotal or attributed secondhand; not presented as controlled results. - **Determinism vs. creativity:** Presenters advocate deterministic verifiers while explicitly rejecting "wrapping determinism around LLMs" — resolved in-talk as: deterministic *checks* on outcomes, creative *paths* to reach them. - **Demo failures on stage:** The live `prove`/saboteur exercises failed or produced ambiguous output; Nick acknowledged, "I need some verification tests around my... verification examples" — recorded as stated, not smoothed over. - **Trust calibration:** Zack on CodeRabbit: "we haven't everyday achieved this perfectly"; Nick: screenshots "might just be like another lie." - **Speaker identification limitation:** Transcript lacks per-turn speaker labels; alternation between Nick and Zack is inferred from self-introductions and content but not individually verified for every turn. # Video Attribution & Provenance Record ## Video Overview - **Title:** Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS - **Format:** Workshop presentation ("Beyond Done") at an AIE event, in New York, with live audience Q&A. ## Speakers & Roles ### Identified Speakers - **Nick Nisi** — Engineering Manager at WorkOS, based in Omaha, Nebraska (~2 years). Co-presenter/creator. Self-identified heavy agent user ("I haven't written code since July 30th, 2025"). Builder of personal agent tools (see Internal Tool References). - **Zack (Zach) Proser** — Engineer on the Applied AI team at WorkOS. Co-presenter/creator. Builder of the **CTX** tool. Note: transcript renders the name as both "Zach Prosser" and "Zack"; title metadata says "Zack Proser." ### Other Voice Sources - **Audience members** — multiple anonymous questioners during Q&A segments; not individually identified. - **Quoted third parties (not speakers):** **Lauren Tan** — quoted via a post attributed to her on X (described by presenters as "a famous potato on X"). Second-hand quotation; exact wording paraphrased/possibly garbled in transcript. ## Referenced Sources & Claims (Second-Hand) - **WorkOS internal anecdotes:** claims about team members shipping "900 PRs a month" with "no incidents"; Applied AI team experiences with eval baselines. Presented anecdotally by the speakers. - **Lauren Tan (via X):** quote on agent self-verification ("if an agent can't verify its own work, nothing else matters"); also cited as among those "achieving thousands of PRs ostensibly a month" — anecdotal, per presenters. - **AMP team blog post:** referenced but title uncertain ("loop-driven develop... I can't remember" — speaker explicitly unsure); example of ball-in-circle feedback loop demo. - **Named tools/projects referenced:** Playwright, GH CLI, Chrome DevTools, DataDog, Cloudflare, Bun (Rust conversion), Theo (Rust conversion, "last week"), Reptile (automated code review, used by WorkOS — claims "5 out of 5 score"), CodeRabbit/Codo (mentioned as alternatives), Jev, Cleft (Cloudflare, "2 days later" after Jev), ChatGPT, Cloud Code, Pi, frontier models named as "Fable, Astra, Saul 6.1, Opus 5.5." - **Incident reference:** an unnamed widely-discussed tweet about an agent "deleting production databases" in 9 seconds — source not specified beyond "a tweet." ## Promotional Segments (Flagged) - **WorkOS-adjacent promotion:** presenters are WorkOS employees promoting their team's practices and tools (CTX, Ryker, snack) — explicitly framed as their own experiments, but self-promotional. - **Workshop materials:** repo and site at `talks.workos.cloud/beyond-done` (speaker initially misstated as `workos.cloud/umbeyond-done`, then corrected) containing slides, exercises, glossary, and an "ask anything" AI agent. - **Booth invitation:** closing promo for "expo hall booth six" to chat with presenters. - **Third-party service mention:** a provider (unnamed in transcript) offering "$5 for free when you sign up" — promotional mention. ## Audience Interaction & Availability - **Live Q&A:** multiple audience questions (determinism/JSON, CI vs. local verification, skills vs. CI enforcement, deployment/preview verification). Audience show-of-hands noted (majority building with agents). - **Resources availability:** repo, slides, glossary, and interactive exercises stated to be available for attendees post-session ("go through it on your own time"). - **Follow-up channel:** in-person only (expo hall booth); no other contact channel stated. - **Comment availability:** no comment section or online Q&A channel mentioned in transcript. ## Provenance Notes / Uncertainty - Speaker name inconsistently rendered in transcript ("Zach Prosser" vs. metadata "Zack Proser"). - AMP blog post title explicitly not recalled by speaker. - Several model/tool names may be garbled by transcription (e.g., "Saul 6.1," "Reptile," "Pi," "Jev"); recorded as spoken, not verified. - Demo failures occurred live (proof command failing, output issues) — recorded as presented. \>\> Hey, welcome. Welcome to Beyond Done. Uh we're going to get started here with this workshop. We got a lot to pack in and uh we want to make sure that we have time for questions and um uh getting getting like everyone situated. So, we're going to go ahead and get started. \>\> \[clears throat and cough\] \>\> So, what we mean by Beyond Done, just to like jump straight into it, um you probably saw a tweet about this incident that happened a while back uh where an agent just started deleting production databases. And in 9 seconds, everything was gone. Um that is what we want to help you prevent here today uh while still trusting your agents and not having to look at every line, even look at every PR. Like this is where we think things are going and how do we get there? How do we make this a reality where instead of having that broken piece, we can have you calm, have uh everything work, and let you to some degree trust your agents, but build that trust over time. \>\> I'd say also that it's not just about preventing errors, right? But this is also what we're going to look at today foundational to achieving true scale with it with agents as well. \>\> So, I'm Nick Nisi. Uh I'm an engineering manager at WorkOS. Uh I live in Omaha, Nebraska and uh I've been there for about 2 years. And I do a lot with agents. Uh I haven't written code since July 30th, 2025. I know because that's the day I got back from Laracon and I saw PHP and I was like, "Never again." And uh here I am. Uh and with me is my best friend, Zach. \>\> Hey, I'm Zach Prosser. I'm an engineer on the Applied AI team at WorkOS. Super excited to be here with you all today in New York. \>\> \[clears throat\] \>\> Now, uh we had this up before. This repo is your workbench. Um all of the slides are in this repo. You can get it from here. You can also view it at um workos.cloud/umbeyond-done. Uh, I know that's a mouthful, but the next PR or the next QR code, I believe, will take you there. Um, sorry, talks.workos.cloud/beyond-done. Uh, and I just wanted to call it out because on that, I'll switch to it real quick. Um, all of the slides are out there. There's also a page for the workshops, like the the exercises that we're going to go through. And so, for each piece of it, you can go through it. You can go through it on your own time. You can go through it later. This is a a step-by-step guide to go through that. There's also a glossary uh, covering all of the terms, big and small, uh, some that you're familiar with. Uh, hopefully most of that you're familiar with, but if there's any that you're not, uh, you can talk in here or sorry, you can look at the the glossary for those terms. There's also an ask anything button, and you can talk to a reg that's trained on this uh, slide deck, and uh, hopefully get your answers there. But, you can also just talk to a human, us. \>\> Indeed. Yeah, we'll leave leave this up as well. We find that a lot of folks want to ask questions and for whatever reason can't, so be sure to check that out and use that. \>\> Cool. Give it just another second. I see some cameras pointed at QR codes. Security would be so mad at us. All right. Um, so when we're thinking about uh, being done with work, uh, agents being done, uh, there's lots of things that go into us trusting that. And easily the first one is just is the code right? Is the code what I would have written? Did it generate a good PR? And then I go through that like a normal normal code reviewer. Um, but that takes a long time. And these agents can write code a lot faster than us, so we don't want to rely on that too much because it's uh, it just there a a signal-to-noise ratio that starts coming into play and we just have too much code to review. We're not writing any code anymore. And we're less familiar with that code, so we become further abstracted away from it. So, we need to find other ways to do that. And there's plenty of ways that you can do that as well. There's you know, using tools like browser tools like Playwright for example, to say like, "Hey, I have this website and I submitted a form. The submit worked. I checked that and it showed me a success message. That's a great test to write. Not even really anything agentic about it, but it is something that agents can use to verify their work. And they can self-report that. And you can trust that often times, but not all the time. And so, that's really what we want to drive is like, how do we make it so that we can actually trust these reports? \>\> Absolutely. So, everything that we've got here for you today in this workshop as Nick mentioned is backed by a repo and it is interactive, so be you know, feel free to take an opportunity to actually run this to experiment with it to take something back that you might you know, use at work or at least experience in a different way. \>\> Yep. In this so like if you just go into the repo like I'm doing and you do PNPM snack if I can type snack up uh This is why I have shortcuts for everything. P is my PNPM. Um it will pop up the browser and we can look at it and we can see this Midnight Snack Club app that we made. Um and you can play with it right now and I can test this myself. Let me just put in my email address. I'll put in um zach@spamhim .com. \>\> Thanks, buddy. \>\> Yep. Uh and look at that, success. It saved the email. The email is here. I can see it. If I had a test that was looking to confirm that the email went on the page it worked great. So, that's what these examples look like. You can kind of see it um, as you're going. Uh, but if you reload that page, just like this, nothing's there. So, even though, oops, what did I do? Oh. Switch back. Uh, if you reload that, uh, it shows that nobody's here. I can't drive Chrome, apparently. Um, and that's because it did update, but it didn't persist that anywhere. So, we didn't have any kind of verification. We're just checking UI, seeing, oh, did that actually do anything? And yes, it did do something to the UI, but we didn't check the other half, and so it was never persisted, and we lost that email. So, sorry, Zach, you're not signed up. \>\> It's okay with me. Um, yeah, I think this is a really critical point. So, uh, just show of hands, who in this room is is building with a ton with agents, just to start with the first question. Okay, awesome, like more than half of the room. Uh, how many folks are running multiple terminal tabs or multiple chat sessions or multiple agents, like everybody that put their hands up the first time, right? And so, this is what we're all experiencing, at least what I'm experiencing, what Nick's experiencing. If this is the 18th, you know, lane of work that you're driving today, and you have like, you know, diffracted attention to actually spend on it, these are the subtle bugs that will creep in because the agent's like, yeah, I did what you said, the code looks good, it builds, it runs, the test pass, and it's still wrong. And so, what we're going to ask and explore in this workshop together is, how do we actually fix this problem so that the agents that we're now all managing as like herds of insane token machines uh, can actually figure out what they need to do themselves and when they've made a mistake. \>\> Yep. \>\> And you'll notice that we are aren't really looking at the code for this. We're not really going to look at the code too much for this because, uh, like I said, I saw PHP and I'm scared, so we don't look at code anymore. \>\> \[clears throat\] \>\> Um, but all of this builds up to this great quote by, uh, a famous potato on, uh, X, call, uh, her her name is Lauren 10. Uh, and she has this quote, if any agent, sorry, if an agent can verify its own work, nothing can't verify if an agent can't verify its own work, nothing else matters. \>\> Yeah, another way to say it is is that and I think this is true in in my experience and others as well is that we're kind of converging on the point that verification now, especially as model capabilities advance rapidly is is one of the key foundational things to focus on, right? It's like reliability and verification and evals. Especially as we're starting to see more and more harnesses come online that have the concept of RSI or you know, recursive self-improvement baked into them. It feels like we're shifting to a world where we may all be maintaining multiple self-evolving code bases eventually, right? And in that world, how do we set the eval foundation so that the agent actually knows what good looks like and can do those hill climbs on its own and then how are we going to give the agent the feedback it needs to understand that it's failed something. \>\> Yeah, until we can solve that, we're not going to trust these agents to really accelerate our work to its fullest extent. And so we need to solve that in various degrees and it's not just going from zero to one, but it's a gradient and getting us closer to one to where we can fully trust these agents is is the goal. \>\> And I'll also just share that you know, again, anecdotally, this is what we're all experiencing at WorkOS. We've got an applied AI team that's building a ton of different stuff for other teams and for ourselves. And it is just a night and day experience when you are iterating on a code base that has, you know, the foundational eval in place, a baseline that can be run and trusted. You know, we'll get into some of the tips and tricks, but the ability to actually see that it did or did not do something correctly in the same way that we would as human developers, right? So we're, you know, of we've the the metaphor of us all becoming agent harness managers is is, you know, well trod. This is how we ensure that each one of them is actually doing correct work and we're not burning money and time essentially. \>\> Yeah, this is the equivalent of saying like make no mistakes, right? Yeah, if you want to have some verification in there, you can't just like guess at what it needs to be. You need to give it clear, defined goals to get to and give it a way to observe and check those goals. If not, the agent's going to do something and it's going to spend some tokens and it might get to the end that you expect it to, but there's no guarantee of that. Every for every detail that you don't fill in in terms of an outcome and what you actually want to get to, it's just going to infer that or make something up and that assumption that it makes may not be the assumptions that you make or that you were expecting because you're a rational logical human and you were just not aligned on that yet. And so um you need to have these systems in place to allow for it to observe, understand what you want, observe that, repair that and then recheck that. \>\> Yeah, and I think just again, anecdotally like we're all finding that you know, the tired way to do it is to have a giant massive PR and have one agent session that's running for three or four days and say fix this, fix this, fix this, make it better. And what is working, you know, essentially perfectly for folks is decomposing tasks really cleanly, giving the each task an agent in the prompt exactly what it needs to do to know that it is actually done and to verify the state of done. And those will just rip. I mean, we have people on our team that are 900 PR's a month and and no incidents and like improving KPIs for the software. So just anecdotally it is um something that we all kind of have to take a step back and think about. I'm not approaching this code base anymore in the traditional way. I'm almost gardening multiple code bases and letting them self evolve. Yeah. \>\> And this is something that we build and we hold our own tools too as well. This is how Zach and I stay productive at Work OS and how we get our own work done. We've each built kind of our own examples of this and our own ways of approaching this. And we really think that that's like a good thing right now. It's too early to really classify on anything, so experimentation is key, finding what sticks and then running with that. And uh one thing that we're both running with is the verification piece or like driving everything from that verification loop. Uh because uh as as it says in this quote here for for mine, uh status comes from records, not model pros. Um so it's it's, you know, show me the receipts. Show me the actual tangible proof that it actually did it uh what I expect. Don't just tell me, "Yeah, it passes, it looks good." Um because there's nothing to back that up. I need actual screenshots, videos, test results, smoke tests, um any all of the above to prove to me without me having to look at the code that it's actually doing what I want. Because at the end of the day, I don't care about the code as much. I care about the result and what it's doing and what value it's delivering. And it can have bad code, honestly, and deliver that result pretty well. Uh or it could have really good code that doesn't uh give me that result at all. And I need to find the right balance in there, which is giving me the result that I want, not the code. \>\> Yeah, indeed. Uh so I've been doing this as well. I I built a tool that I call context or CTX. It started out as a cross-agent harness memory layer cuz I'm rapidly experimenting, as all of us are, with like ChatGPT, Cloud Code, uh Pi, etc. And I wanted them to all be able to write and read from memories, and that started working. And then the more that I was running more more multiple agents and, you know, average day is like 18 or 30 terminal tabs, the more that I realized uh it might be an interesting experiment to try to centralize authority for verification in a web service. So the CTX is a simple web app that runs on Cloudflare. It was vibe coded initially. It works. Uh it started as memory. It was able to hand out secrets to agents without them ever seeing it or getting the credential. And that worked, too. And then now as an additional experiment, I'm creating verifiers. And the way that that works is the agents are assigned work, as we're all doing with prompts. They go off and do that work. They come back through the MCP connection to CTX and say, "This is what I've completed. Uh am I done? Please verify it. So, let's say in the example of um uh one of the workflows needs to verify that a PR was created correctly. And so, that is going to run an evaluator on a known verifier. That could just be a simple CLI to start with as it is in my repo, and then that becomes a hardened piece that you can reuse again and again. It's going to the GitHub API and verifying that that PR truly does exist and is healthy and that all the merge checks pass. And only then will it mark that node satisfied in the workflow. Otherwise, it will redirect to it will redirect the agent and say, "These are the things that are still failing." Right? And so, again, an experiment, but this is kind of how we've been trying to play around with this and see which patterns are are going to stick and and what works and what not. Um but it's also just kind of a fun time to do that five coding because uh I require uh weird pixel art across all of my apps. So, uh you can see in this case there was a simple workflow. And one of the verification gate is that requires that all pull requests were merged. And so, that work is done by the centralized authority. It goes out and uses its verifier, checks the API or whatever the artifact is, and then finally delivers the status over here on the right sidebar if you go to even down, and references the actual repository checked and the PR, right? So, simple example, but you could imagine having standardized verifiers in a library or in skills or in tooling that you share amongst your your org, and this could be one of the ways that you start to build toward accelerating, you know, PRs and and less and less human oversight. \>\> Mhm. \>\> Uh and then a similar tool uh that I built is called Ryker. This is more like a engineering-focused Grokbot or Muse uh that's trying to be extremely flexible in how you hand work to it, but then once you hand work to it, its job is to babysit it. And so, I don't want to have to babysit all of the PRs that I'm I'm bringing to fruition. At some point there's a like this level of diminishing returns for me and my focus on on the work that I'm doing because I end up just like babysitting the PR, you know, fixing the tests, fixing the the reviews uh that are coming in, the the review comments and things like that. So, I want to be able to get the idea down of like what we should be doing and how we can verify that that work is done uh and then I can hand it off to Riker. It can go do the work of, you know, making sure that the the reviews are done, uh that when it fixes code uh from PR reviews that it's still matching our overall goal of working software uh in some way, and then uh can just verify that and and alert me back when it needs me instead of me having to baby sit it. Um and so, with that it's just um you know, it builds a check, it figures out what tools it needs to run, it will run them, uh verify that they all pass, it'll keep that evidence and show it to me in different ways, uh and then decide when to stop. So, if it can verify yes, it was working, we've now fixed this bug, for example, um we put them together and it works. Uh so, it's really just keeping me out of having to be in the minutia of shipping the the like final 10% uh and just focusing on where I'm valuable as a human, which is why we're doing the work and how we're and uh to an extent how we're doing it. \>\> Okay, so maybe just as a mental experiment or a thought experiment, if you will, let's just kind of show of hands like or feel free to shout out like how would we start to think as a human developer of verifying a web app like maybe in the before times before an LLM. \>\> Yeah, what are things that you would do to verify? \>\> And we'll we'll throw some easy ones up here to get it started. So, like refresh the page, is the application still visible, right? What's that? Absolutely, yep. Open a new tab, see if it's healthy, right? What are the response codes I'm getting back from the API? Are they expected based on what I sent it? Absolutely. \>\> Local host or local storage, checking persistence in some way, console logs, um things like that. Network, uh, like you said. And, um, like reading what's actually on the page. Do the contents of the page actually \>\> Interacting with it, right? And the reason we're we're saying this is like this is really to set up the the transition from we were the human developers doing this all the time. This was part of our core responsibility when we're hand chiseling code, if you will. And then, what is the delta between doing this and then in the the early stages of LLM time saying, "Here's three paragraphs of text. Go execute these instructions." The delta between those two things is the agent's ability to verify that it actually did what you asked it to do in a real application and that its changes did not break anything. Right? Otherwise, you're just getting the confabulations that LLMs are designed to give you cuz they're they're trying to process the text. They're trying to make you happy. \>\> Uh, your agent can't do any of this just in a terminal alone. Uh, it needs tools to do that. Uh, fortunately, tools are very easy to add to \>\> agents now. \>\> Um, and we just have to tell them how to use it. We'll tell them specifically what we're trying to to prove. Um, when you do that, you don't want it to be just a paragraph of like, "Hey, do this very always verify the change in a browser." Whatever that means, that's pretty vague. Uh, "Remember to reload and check." That's a little bit more specific. Uh, "Check that the data persists." Look at the console for errors. Uh, getting a little more specific, but like that's a lot, um, to put in there and you can certainly do that. That's what a lot of skills do. That's what what a lot of just like saved prompts do. Um, but if you forget anything or things change, that can leave it up to a lot of interpretation. Um, one thing that might be better is to have a script that can prove it out in different ways, um, so that you can just run that over and over and it's doing more of a deterministic check across a number of things and then it can feed that output back into the agent and it knows exactly what was wrong uh, and how to fix it quickly. And it just closes that loop so that it's a much faster uh and the agent can just spin that, run it again, check, verify, run again, and just do that over and over until there's no more errors. \>\> It's also uniform if you think about it, right? Um we know that LLMs are non-deterministic, so even giving 30 agents the exact same prompt and you have a potential for variance in your outcomes. And pinning it down to a single command that is backed by actual bash commands and bash calls, right? Like that's that's significantly more deterministic. And anecdotally, the folks uh like Lauren Tan and others that are achieving thousands of PRs ostensibly a month and uh building ambitious products that were not possible to build in that time before have mentioned that this is the first thing that they they concentrated on that actually unlocked the velocity, right? So, if your specific web app, for example, does not already have a bespoke CLI that allows any agent to fully flex the functionality that you care about verifying, then you're going to have a much rougher time and an unpaved path if you're using prompts, you know, alone uh compared to actually, again, giving each agent the ability to say, "Yes, I actually built that feature. It is actually performant. It didn't cause any errors, and the application looks correct." \>\> So, for this workshop, uh we made a little tool. Uh we've been running it a couple of times, but it's called snack. Uh and you can just run it with PNPM snack in that repo. Uh it's not meant to be like a reusable tool, but just kind of like a template for how you could set up this uh specific for the applications that you're working on, and then use that as a way to tie into the LLMs that are actually working on that application with you. And so, snack can do a couple of things. It can uh manage the health of the application. It can bring it up, it can take it down. It can run a doctor command to see is it actually, you know, is the database running? Does it have network connection? Et cetera. Um that quick drive or that quick check that it can do. Uh then it has uh drive, so you can tell it to join. Remember, this is our our uh midnight snack waitlist. Uh so, it can join the waitlist. It can reload the page, it can reset all of the data so it can't find it. You can have it look up things, so take a snapshot, uh, look at the network storage, look at console logs, um, look at local storage, uh, screenshot it, run evals, all of all of the above. Uh, and then finally it can prove it in two ways. It can actually run a prove command where it's checking for deterministic output of that. And a judge, which is using a system one model to, uh, in like, get an idea of whether we're going in the right direction without it necessarily matching one to one. \>\> Just to, like, fully develop this idea, when you do something like this in a repo that you haven't already had something like this before, the velocity gains that you unlock are not solely for you, right? It's for everyone else on your team that then comes along and is also able to issue significantly more PRs significantly faster and significantly safer, right? More safely, so that, um, you know, everybody on the team actually benefits. And again, that's like where we we keep harping on this being like a foundational concept now. \>\> Yeah. Uh, and if you think about it, like, that's a really big piece of it it might even be more important than your software, uh, than the actual application is the way that you verify it, the way that you test it, uh, the way that you prove that it works because that is what you could use potentially, like, a lot of people are doing and converting to Rust. Um, you could take your app, uh, like the Bun team did, convert it to Rust, uh, Go, TypeScript, convert it to Go, uh, not necessarily with the same level of LLM help, but, um, they did use LLMs to help with that. Um, and then I think Theo just last week converted it to Rust again, so. \>\> Yep. \>\> Uh, when you want that, if you have all of those verifiers in place, then your source code doesn't matter as much. You can write it again and prove that it works. Keep going. \>\> Absolutely. \>\> And so, um, these tools give your agents their eyes. Or you if you think about it, like, the tools that your agent use, that gives it the body. The LLM is the brain and then the way that it reaches out and interacts physically, uh virtually on your computer is through these tools. And so, things like uh join is it like actually having it run and do things and then it can check and see, you know, did I did I that actually succeed in some way? Uh check storage, run reload, uh snapshot, etc. And so, let's just run a couple of these uh and see. P snack up. That brought up our um browser. And then I'll open up another browser or another terminal. I have too many terminals. And run um P snack snapshot. And that give us a snapshot of the page in as JSON that we can see. If I put um Zack at zack.zack uh and add that to the page. Hopefully, we'll see some Zacks in there. There we go. So, that's a way that we can verify. Uh we can also do screenshot to make it a little easier. That should pop up a screenshot. And yeah. So, that gave us the ability to, for example, not just like pass this to the LLM and have it do interpretation on it, but like proving back to us. That's the end goal, right? Is proving back to the human. Being able to take screenshots, record videos, put those on the pull request. You can do that with the the GH CLI now. \>\> Um Chrome debug debugging tools. \>\> Chrome debugging tools, yeah. \>\> Where is the actual bottleneck? Don't guess and, you know, spin your wheels for 3 hours. Like actually figure out which part was causing the render to be slow. \>\> Yeah. So, the real key here is that there's no um it's it's not just code that you can look at to interpret whether this this application is running correctly, but you can build tools that will help you prove to you that it's working correctly without you having to read the code at all. Uh because this is all we care about. Is this what a user sees? Is it correctly rendering in this browser at this width with dark mode, with light mode, etc. And you can run infinite tests on that and run them over and over and over in a loop, uh which as Zach said unlocks your whole team to move faster because it you've given the agent the ability to run those checks itself as well. So, \[clears throat\] if we run this uh if we ran it just like this where we ran the doctor uh and then we add uh ada@example.test and then we check the storage, it's there. We run uh if we check the storage, it's not there. Um we would see local storage is empty. And if we rerun reload and then snapshot, we would verify that it is in fact empty. So, this is a way that the tool itself could run in a loop, could check all of these things, and yes, confirm with before it even gets back to us. Oh, I see that we're not actually persisting this. I forgot to do that part. And the agent can just move on from there. Um real quick, we wanted to stop and take a a few questions if there are any. \>\> Yes, sir. \>\> Yeah. Uh so, the question is uh are we uh using models to write JSON uh to to then like help us to make our code deterministic, to bring that back. Um the answer for me is yes. Uh I'm I'm using that and I'm running it and then I use uh like I'll I'll use unit tests around that to verify in different ways, catch edge cases, uh and then verify that the code that that is like a real interpretation or real representation of what I'm expecting to get back from like a system one model, uh and use that to prove that that I'm on the right track. But, then have keep those as well to kind of like have a another check uh to make sure that it doesn't drift over time or things change. \>\> The the the general process that we're talking about? Yeah, so when what I'm talking about, uh, at least on our team, we we build a ton of applications that are internal tooling. We have some customer-facing applications as well. They tend to be uh, TypeScript and they tend to be deployed to Cloudflare. So, they're you know, they're full-stack applications. It works perfectly. You can even have evals that are focused on the back end or database operations. All right, you could get as granular as you want with it. Um, but it's absolutely yeah, it works full-stack and and I expect that the way that we're all going to continue interfacing with software in the future is just going to be kind of like the gardeners making sure that they're evolving correctly on our own. Yeah. \>\> That's why uh, we're not like giving out a tool because we couldn't give you a a tool at this time that works generically for every single application that everyone in this room is working on. You kind of have to build up the tools that fit the app that you're working on and then give that to the models and just make sure that they know how to run them well. Um, as a \>\> I was so I was I was also going to say like your evals, what you actually need or care about as a team or company or business or going to wildly differ from, you know, if you're in like the interior design space versus some other place where like speed is the only thing that matters. Um, I I think you're going to have like wildly different evals and there's a value to building those up with your team. \>\> Yeah. And there's there's lots of different ways to get the data to the like to the model in different ways for so that it can interpret it and it can keep up. And that's like a a huge piece of like keeping the model in the loop as to what's going on. Uh, the AMP team has a really great post on it. I think it's called I can't remember the the post, but I'll get it out there. I think it's called loop-driven develop I don't know. I can't remember. But uh, what they in in the example, they show like a canvas with a circle and a ball bouncing inside of this circle uh, and then like there's a little slit where the ball if it hits just right, it can fall out of the circle and then the game the game is over. And uh, the whole idea of that is making things feedback loopable. That's what it is. Um, where you know, the model can't actually see the running canvas and where things are going, but if you, as a very contrived example, if you just print out the coordinates of that ball at every point, and you see a bug where, you know, once it gets to this specific point, it jumps to the other side of the screen. Well, if it's moving more than one pixel at a time, like per loop, something is wrong, potentially. And so, there's a bug in there. If you can surface that to the model, then it can help you debug it, where it doesn't actually have eyes to see the canvas and see the ball not behaving well, like with physics. So, \>\> And also, once you do that, you can you're also then free to spin up 30 different variants of software that are, you know, speaking to that eval, and then pick the one that you want. \>\> Yeah. \>\> Um, so, that's the other thing that it unlocks, right? It's really like foundational for just going, and especially if we expect agents to models to improve, you know, tokens to get cheaper, potentially, um, then, you know, you can imagine that we're all going to be like coming in on Monday and saying like I would like to see 35 versions of this new feature or application here. \>\> Yep. \>\> And pick one. \>\> And so, how do we actually run or create that? Well, in this case, like we said, we can't really prove it, uh, for everything, like universally. It really has to be something that's unique to your application. And so, for our midnight snack page, this is what the proof, uh, the proof command might look like, where it's basically using, uh, a tool like Playwright, um, to check that a, uh, an email was actually saved when it gets on the waitlist. So, uh, we can check that it doesn't have any on the waitlist, any emails, then we can fill in an email on the next line, and then we can get the button and click it, uh, the join waitlist button, and then we can expect that the status, uh, element on the page has a waitlisted in there, uh, and then check the inner text of that to see if our email address is in there. Um, and so, that's one simple way to do it, very easy if you've ever written, uh, an end-to-end test with like playwright or react testing library, any of those, you've definitely used tools like this. You can just give them back to the model to your agent and they can run them. Give it to them as a tool and they can prove it out themselves and you've made this reproducible bit of evidence that then you can use in this one next time you use it next time that page changes it can run it again and verify that nothing broke. It acts as a regression and keeps everyone on the team moving forward. \>\> \[clears throat\] \>\> One caution word of caution though, like a successful screenshot isn't proof. Make sure that it's actually showing, you know, the state that you expect it to be in, that it actually like ran the commands to get to that state to actually be in there because otherwise it might just be like another lie, but you're seeing a screenshot on the the PR and thinking, oh yeah, this is this is correct, but it might have be causing different side effects. It might have taken a different path that's not actually possible within the UI. So you want to make sure that it actually like runs start to finish to check that. \>\> Yeah, and so once you build these loops that are, you know, durable, reliable, repeatable, uniform, the value of what you get or the benefit the main benefit is that a a failure starts to drive future work, right? So if in one loop the agent fails, you're busy talking to somebody at work or you're starting yet another task, that agent can determine on its own, this is not quite correct. Let me go take another stab at it. Let me now run the command verify again. Okay, now we're good. If you think about what that means at scale that a single PR or an agent a single lane of work, if you will, and an agent, you know, self heals, self corrects, figures it out on its own, you're not coming back to that particular PR at the end of the day and and then being like, oh, I still have to get this all the way through. And if you just multiply that out, right? That that's that's really where again, we just say this is foundational for speed and velocity. Yep. \>\> And how many of you write good error messages? Show of hands. \>\> Yeah, nobody knows. We already know. \>\> \[laughter\] \>\> Sir, what's your message? \>\> I got small hands. \>\> There you go. Yeah. \>\> Solid. \>\> Um well, you should. Uh I think they're more important than ever because uh when your your uh app throws an error, guess what can interpret and run with that error really well? An agent. So, um if you make it so that your failure is explicit about why it failed and give as much detail as possible about how it got into that state, then uh your agent can just like self-heal from that. So, it's um very easy, very worthwhile to to do that. \>\> Uh yeah, and so this is a concept that you know, Nick and I talk about, right? And it sounds like we're being uh in some ways we're trying to wrap determinism around LLMs, uh but we actually don't want to do that. And it's also something to the spirit of what the amp team is talking about. Uh the agents are in in intelligent. They're they're capable. We want them to be creative, actually. Because oftentimes if they have the right constraints, and then oftentimes constraints can be can equal freedom, right? In a way. Uh then they will come up with creative solutions and actually solve what you said. That's why you give them the outcome. That's why, you know, Nick's project he's experimenting with is outcome-driven development. Um and so, it's not about over being overly prescriptive in how the agent achieves things. Um that can actually lead to lower performance in some cases. It's more about giving it the map and saying you must get here. You'll know you're here when this command returns this. \>\> Yeah. You don't necessarily have to prescribe how it gets there, though. And this is true more and more true every single day with like the frontier models. Fable, Astra, Saul 6.1, uh Opus 5.5. They're all really good at being like you being creative is like the way I I describe that. But, if you give them This is where I want to get to and don't necessarily tell them the path that they need to take, they might wander and make sure that they catch more edge cases. They might figure something out or ask you a question that unlocks something. But, if you're just like you have to get here and then here and then here, they may not sway from that. And you want them to sway a little bit, but get to the right right end goal. And like I said, this is something that we we've only come around to really appreciating in the latest batch of Frontier models. Uh where it's really becoming possible. So, going forward, it's going to just get better. \>\> Yep. \>\> \[clears throat and cough\] \>\> So, when it's going, um you want to get into like, you know, the typical TDD like red, green, refactor loop with your agent. Um we've talked You've heard about loops, you know, at every one of these AIE events, and um it's no different here, but you can use that within the verification piece itself. Verify this, reproduce it, show me uh fix it, and then show me the proof of that. And how are you going to show me the proof? That's just as important as the fix that's going to go in, because that's what's going to leave leave you alone and allow you to trust the model to actually do the work. And And then when it says, "Hey, I'm done," you know that it's done because it's proved it to you in some way. \>\> Absolutely. \>\> So, in this example, uh you can run Snack Doctor. Let's just go ahead and run that real quick. P Snack Doctor. And so, this uh this ran, and it's showing us um you know, a bunch of checks that it did. So, it's checking that we have Node, that Chromium's installed, uh that our session exists, uh that it's actually running, and that it successfully built. So, that's like what we're running as a quick doctor check, just a bunch of different things. So, then we run the next command, Snack Join Ada at example.test. \>\> Sorry, we don't type anymore. \>\> \[laughter\] \>\> Okay. Gave us the result back. Snack storage. Storage is empty. It didn't actually put Ada in there. \>\> So, this is the point at which in the real world your agent you've just gotten the value out of this entire system cuz at this moment your agent realizes I am not done and I'm not going to drop this work on the floor. \>\> Yep. And if you reload it, do the same thing one more time. Boom. We have now given our app eyes. It can see this. It can run these. It can make it can make sense and change the world of the application, the Midnight Snack application, and see the results of that and then be able to interpret what happened from there. \>\> \[clears throat\] \>\> So, you can try that. You can If we like apply the persistence fix to it and add honest copy to it, then we can um run the proof. Let's run that. And that kicked off a browser and it failed. I didn't actually run the test, but if we had run the fix. If we had run the fix, it would show that. \>\> \[clears throat and cough\] \>\> Okay, one of the things that we're we're thinking about a lot lately and talking about and starting to roll out into more applications is this like concept of receipts. Um you know, you can you can take like a ticket like a a help ticket system where it's there's a certain performance the way that a human would answer a ticket and consider it done well as an excellent, you know, customer response or service. And then what does the um like preserving that in some way, for example, right? Uh why do we want to do that? There's a ton of reasons and we'll get into some of them, but again, if you think about if it if we all or if it seems that in some way many many folks are heading towards, you know, running RSI harnesses and self-evolving code bases, everything that we do today, everything that your agent does today, every run all day long, that is very valuable data. It's very valuable data right now because you can feed it into LLMs and and get fixes that you need. It's incredibly valuable data in the future as tooling improves and it gets easier and easier to set up RSI loops because then you've got a data set of exactly how your system performed that you can create, you know, very comprehensive evals for and have agents hill climb on, which means loop and see their performance and then continue going until they get better and better and better, right? \>\> Yep. And at that point they're able to pass on their own using that evidence from the previous runs. \>\> \[clears throat\] \>\> So now, if we fix that code, it now persists, but it can also lie. So after we reload the page, it could say wait listed, but what the screenshot might actually say is you're in, your seat is confirmed. Those don't exactly match up, and so that might be like a failure right there because they don't exactly match. So this is kind of getting away from the static testing piece of it. Like how do we prove that it worked without having to be like very tightly coupled to the actual text on the screen for a very contrived example. And so if we just run like, you know, P snack, judge, should run that and it failed because even though it worked and it says that we're in, Jev did not the the model did not see that. And it's because the the confirmation text breaks the contract. It's not exactly one-to-one. So how do we fix that without adding a lot of tight coupling to everything that we're doing. The answer is those system one models, which we'll get into here. \>\> Yeah, absolutely. So, just to introduce it for anyone that hasn't heard of it before, the you know, concept of system one, system two comes from psychology actually. It's like a useful model to think about how the human brain works and how he thinks. So, system one is super fast decisions. It's It's, you know, autonomous. Like, someone swerved at me while I'm driving and I just corrected without even thinking about it kind of thing, right? Some I was about to lose my balance and I I corrected it. And then system two is like I need to figure out what I'm going to do with my life. So, I'm going to go book an Airbnb and like think about it for 3 days. \>\> \[laughter\] \>\> And so, you know, applied to the world of LLMs now we've got Jev and then immediately after we've got Cleft from Cloudflare, you know, 2 days later. That's even faster. And so, there's It's It's a really exciting time to think about all the ways that we can combine these tools together to make the new primitives that we're going to continue to see in evolving AI architectures. \>\> Yeah. To get Put it another way, that light right over there, if I said, "What color is that projecting?" Quickly you would say yellow. That's like system one thinking, right? You just You know the color. If I said, "Why is it projecting yellow?" Well, because we're thinking. Now we're thinking, you know, we could go down like a physics of why the light emits yellow. \>\> Why are we here? \>\> \[laughter\] \>\> That's how we got to really what we want to talk to you about today. \>\> \[laughter\] \>\> So, in this we've got another game that we can play within within this. It's a quick game with the snack. We have saboteur and a detective. And so, in this the the saboteur will pick a snack. It'll put a secret bug in there, randomly putting a bug in there. We don't know which. It'll just be random. And then you, the detective, find it with with snack alone. And then prove what that is. Name the bug and give give proof that you found it. So, this is something that you can do if you've pulled down the website or pulled down the repo, um you can just run pnpm snack up --bug random and uh give it uh let it run and see if you can find it. So, let's try that. We'll just give it some time to go. Uh what was it? --bug \>\> bug random \>\> Oh, it's already running. Let me kill that. Kill the right one. Okay. So, \[clears throat\] in this uh it ran uh it's got the app URL, it's running, it's not headless, so we can see it. The bug uh says hidden until you stop up. Yeah, right. Um leave this running in another terminal, run pnpm snack doctor. So, let's run that. All right, so that ran the whole thing and we can see node's there, chrome's there, it's got a session. The proof here, uh the last receipt says fail, but the source is now uh that proof is for code that no longer exists, so it wasn't able to prove that. That was a previous run. Let's run it again. Same thing. Uh can anyone tell me what the bug is? Any ideas? Wrong email? Yep, exactly. Port's wrong. Uh I think it was running at 62174. No, that was right. Let me try it one more time. We got less than a minute. Uh, for that one, yeah, you do. \>\> The tools make it optional though, so you can run through all the exercises if you don't have it and it'll just go, but um, it's also good opportunity to experiment with one if you do. \>\> Yeah. \>\> They're they're pretty cheap calls. \>\> Uh, they give you $5 for free when you sign up and I think that'll last you like 3 years. \>\> Yeah. \>\> All right. \>\> So, \[clears throat\] um, yeah, go ahead. Next one. Yeah. \>\> Um, so, yeah, we ran it a couple times. I think there was some problem with my output. I'll debug that. Uh, it's funny, I need some verification tests around my verification my contrived verification examples. Um, but the, you know, the the outcomes that could come up are uh, that it named the bug uh, with output, uh, said it works but it didn't, uh, said it works but it did, uh, blamed a bug that wasn't there, um, and then had no, it was the right bug or no evidence. \>\> Right. And this is just getting back to that point of uh, a contrived example, of course, but your team might score differently or have different criteria that you're looking at. This is super flexible. The most important thing to do is to start and to create one, create an eval baseline and to use it consistently and to evolve it together as a team, and then get to the point where your changes and your PRs automatically flex that eval so that you can get to the point where you know if changes are safe to land or if they're improving things or if they're making things worse or if they're not changing things at all. \>\> Yep. Uh and every time it runs and it misses, um that's a new line that can go in the map. So, it can give us ideas of like the steps that we're getting there, why we're getting there, and kind of like you know, every time it fails, it's getting us one step closer cuz we can fix that. So, the first time uh is memory only, um like it's not storing anywhere but in memory. Uh we're putting in a wrong email address. Uh we've lost our confirmation. Every time we can prove that we've caught that and then we don't run into that again. Um but then the over promised one, uh the screenshot or the judge running uh can tell us that it um it actually did pass when it didn't. Same thing with console error. Uh there might not be anything in the console, um but it's in the receipt or it might not be it might be getting caught um before it gets to the console. Um yeah, anything like that. Or it could just be none. There's nothing to find. Everything works. Um and that's a real answer as well. Um yeah, if you run the doctor, kind of like what we were doing, it was showing this where it was showing the previous uh examples. Um it's showing proof that no longer exists. Uh that's because your code might have changed quite a bit between yesterday and today. Um it might be different circumstances, like a a famous date bug, um for example. It might be a number of different things. So, uh we want to build the tools in such a way that they're flexible enough to uh not get caught by um issues like this. Uh this is all kind of like going back to you know, basic unit testing examples, uh but bringing that kind of like deterministic check to the models in a way that they can self-heal and self-improve from there. \>\> Absolutely. \>\> \[clears throat and cough\] \>\> Okay, so this again to tie it back to what we used to do as human developers, right? What we used to be able to see, how we would flex the application, how we would check everything. Uh, that's really what we've been talking about and moving it into code. So, by moving it into code and making it available in a deterministic CLI, then we can get to the same path and the same experience that we would flex as, you know, developers when in Chrome trying to fix something late at night. And that's really what unlocks the the speed and the evolution and the ability to scale up the amount of code that you're delivering and to do that in a more safe way. Uh, again, you know, if you uh, the recommendation is if you have a web app uh, and you're not already you don't already have a custom verifier CLI, that's one of the the highest leverage things you could start with for your uh, code base, the foundational eval as well that shows what good looks like in your application. And then also giving your agents the ability to see all aspects of performance and uh, networking and and traffic and and rendering, et cetera. And so, giving it Chrome dev tools if you're in um, if you're in the web app land. If you're shipping native clients if shipping iOS, you know, or OSX apps, then there's an alternative uh, projects and more coming out every day that you're then going to attach to there. But the the the common, you know, foundation here across all of them is that you need to give your agent the ability to see absolutely everything that you would normally see as a developer. \>\> Yeah, the more that you can give it, the more arms or eyes that you can give it, the more it's going to be able to take over from you. Uh, and then you just have to give it the intuition about when to use those, how to use those. Think about the way that you decide, okay, if I this bug came in, how would I reproduce this? Think about that and if you can break it down and this is dependent on your application and the type of of code that you're writing, um, you can you might be able to do that in a way where it can like think, "Oh, you know, for us it's a lot of sign-on things. Like, is it properly setting a cookie? Is it setting an expiration in that like all of that?" \>\> the in the job, right? You can go again like the this is applicable to every, you know, field and industry that you've got software for. Um but it's it's really just all about like, "Hey, that world where it made sense for us all to hand-chisel code all the time is at least evolving or at least changing rapidly. What are the foundational like what's the what are the same system fundamentals that we bring to this now so that we can accelerate software development?" \>\> Yep. Uh so, yeah. Doing expensive call stacks. Uh but the the important thing is that uh getting that output, making sure that it's giving you detailed output about what happened, uh the state of it, uh all of that so that it can be audited by the application uh by the model, and then you can run it again, it can fix it, run it again, and verify that those changes actually took hold, and that there's no longer an issue. \>\> Uh and yeah, and again uh can't overemphasize the value of keeping receipts. So, if you have runs, uh you know, it can be the entire user journey. Let's say it's an e-commerce app. It's like when they log in, they hit the page, what happened, what was their flow? Did they see an error at all? Did they see 7 seconds of time to see your catalog, you know, come into view? Uh were they able to put something in their cart? Was there cart abandonment? Um and so, tr- save that all now because every single one of those runs and receipts can be fed to LLMs, or you can even set up a system that's running all night long and is looking for uh the latest receipts being produced, and is aggregating them, and saying, "Well, today we saw eight failures in the caching module. So, I'm obviously going to, you know, issue a PR tonight to fix that properly." And then, "Oh, I have the tools I need to prove in this PR that the issue I was trying to solve has been fixed." Uh and then the recommendations that we're seeing from from folks, too, that are achieving like super high scale PR uh shipping without errors. They're they're also able to, you know, add uh the stuff that is for us as humans in our oversight still. So, not just screenshots of the feature, but like a recording of I walked through the app this way. This was my flow. This is what I experienced. These were the states and I got to the goal. So, that if I'm waking up and trying to rip through, you know, 15 important PRs in the the first thing in the morning, I can first start watching that and make sure okay, it actually got to the end and worked. Then I can, you know, start looking at code uh if it's something that I'm actually going to like human review. And uh again, all of that I think is going to be immensely valuable as we continue to see uh evolve self-evolving tooling, really. And you know, think about what that means though because when we get to the point where there's self-evolving tooling is just like the default and people are using it, uh there will be a specific advantage for folks that have preserved uh clean data sets of things that they can say this is what we are trying to get to in this application and want you to get all the way there and rapidly improve it, right? If you're not sitting on a mountain of that and your competitors are and they have similar apps, they're going to be able to go significantly faster in their development cycle. \>\> Yep. And this is just building up that trust so that we can continue to improve upon our app as we go, uh but we have these checks for everything that came before it. So, we've got unit tests, we've got evals, we've got smoke tests, we've got whatever our application needs and we've got a way for the model to be able to interpret and run those and improve them uh in such a way that if we add new features in, we can run all of the previous ones and make sure that we didn't actually break anything, uh no regressions, and we can just mount and climb that evidence ladder, building evidence as we go, uh and ensure that our app is working uh exactly as expected. And that really becomes the the spec that shows what our app does. Okay, cool. So, we've got uh just over 5 minutes. I'm going to pause and see if there's additional questions. Yes, sir. \>\> So, does the scale I I would personally I'm curious with how you think about it. I would I would personally almost lean away from skills cuz in my experience it's it's you know up \[clears throat\] to the model whether or not it's going to actually follow that path and use it versus like having the CI actually enforce that these these commands get run as well. Yeah, that's what \>\> Like the answer or the question is how do you run these these checks normally, right? Effectively. \>\> \[clears throat\] \>\> Yeah, it can you can run those you can run these regularly. You can have them be like part of CI for example. You can have them be various hooks that could be like a get hook although that's probably the last one I would reach for. We could have it be like a cloud code hook. It depends on what your team is using and how they're they're doing it. I might build like a full pi extension to keep things running in the background and just kind of like feed in information as it's going. There's a number of different ways that you can do that but the easiest way by far to get everyone across the board and I can say this now just put an agents.md not a cloud.md in your in your app and then have it say like you know on change verify things by running this or like every time you change significant things or before you open a PR do this. Like whatever wording that you want to put in there and then that's an easy way to get everyone across the board and they really kind of have to go out of their way to ignore that or avoid it. \>\> Hooks are also relevant too if you're doing the local development, right? For the whole team cuz you can share hooks across the board. Yeah, absolutely. Yeah. \>\> Well, CI is a is a I think a feature of it, but it's not like the the actual tool is like the rapid development that you get locally. So, I give my agent a task to do and I can just quickly run uh like let the agent go. You know, I basically tell it this is where I want to end up. I want the waitlist to be I want to be able to add an email to the waitlist and then it gets persisted uh and saved and I can verify that. Uh so, this is how you verify it. It goes into this table, blah blah blah. Like whatever it might be. Um Yeah. No, good question. Sure. Yeah. Yeah, what's the benefit of running it in CI versus like a local Sure. Uh the benefits are definitely like a faster feedback loop, which allows us to run faster and allows me to uh parallelize myself because I'm no longer working on just one thing at a time. I'm working on at least right now I have three agents running. Uh that's a very low day for me, but I'm at a conference. Um \>\> \[laughter\] \>\> Normally, it's like 10 to 15. \>\> He's really taking it easy today. \>\> Um and and the the whole idea is like I'm going to set it off and I want I don't want it to be waiting on me and saying, "Yep, I did it." and then I go running and it's like, "No, that's not actually correct." and then I have to like set it off again and I go get distracted by another agent again and I just continue that loop. I'm just like constantly moving between projects. So, I want to give it the tools to where it can prove to itself that it did it and then prove to me that like show me the evidence in a way that I can trust uh so that when I come back to it it might be in 5 minutes it might be in an hour or a couple of hours I want to know that it has done the best possible job that it can to get done to a state of done so that I'm not just like re-prompting it and starting that whole loop again because I don't want to be that's that's not the fun part of my job. \>\> Yeah, I would I would say another way to to put it is that and we haven't everyday achieved this perfectly but I want to have all of the different lanes that I started that day for stuff that I felt was important to ship that day. I want to come back to them at the end and see them basic like correctly done. What I don't want to do is have a whole long day and have them running in the background be doing other stuff and going to meetings and then come back and then actually have to turn by turn nudge any of them because that is like a significant slow down and it it prevents me from going faster. \>\> And another way to put this is I'm like I rely on CI and the tests that are running in CI and the code reviews like we how many people do automated code reviews with like Reptile, Code Rabbit, Codo, any of those? Yeah. I rely on those. We use Reptile. Reptile gives me a X out of five score and if it gives me a five out of five and all of my CI is passing I feel extremely confident that I'm not going to break production when this gets shipped. We've gotten to that level of trust with Reptile. The more important thing for this is that I get to that state faster and that um \>\> with less turns \>\> With less turns, yes. And that you know, I I might add a whole like I could add 10,000 lines of absolutely nothing and it does nothing and great it passed. Reptile gave it a five out of five for some terrible reason and then it goes into production and it doesn't do anything it doesn't break anything. Now it's just an extra 10,000 lines of nothing. Um and so this is also to help me make sure that like I'm I'm telling the agent like exactly what I want and it gets there without going through a bunch of um additional code to or additional like like it stays on track uh but it also knows how to fix itself to keep itself on track while I'm off doing something else. \>\> Okay, we might have time for one more quick question, but I think we're Yes, yes, sir. \>\> Yes. Yes. Yeah. You could also You could also do depending on your stack like there's it's also you know there's various providers that you could easily also go to preview first and make and then verify it there in a different way. So we'll then go to prod and then also immediately monitor. It's another pattern we're starting to see emerge is that when folks have a deployment that was mostly agentically implemented, there's also at least one agent specifically concerned with that production rollout looking for watching all the metrics, watching DataDog, watching feature flags, right? Being able to say this is this is actually bad and we're we're reverting it. Um so I I think that like I I truly understand and get the like queasiness around it like we all have or dealing with that, but I I I do see a path for that um because it just seems like it seems inevitable. Yeah. Yep. \>\> That's all the time we have, but if you have more questions we'd be happy to chat. We'll be in the expo hall booth six right to the right when you come in. Come find us. We'd be happy to chat to chat, show you what we're working on and talk to you about anything you'd like. \>\> Yeah. Thank you so much for coming and your attention and everything. Thank you so much. \>\> \[applause\]