Talk by Lawrence Jones, founding engineer at Incident.io, on using AI to debug AI. Incident's AI SRE runs many prompts per investigation across logs, metrics, traces, and code; when it produces a wrong root cause analysis, the full trace is too large for a human to inspect. Jones describes three debugging patterns the team built: a small CLI that lets coding agents read and edit oversized eval YAML files, enabling a red-green runbook where the agent writes a failing eval, fixes the prompt, and verifies nothing else broke; serialization of every UI debugging view as a downloadable file system that can be loaded into a Claude Code session so an agent can trace the prompt hierarchy and identify the exact prompt to change; and fleet-scale analysis where 25 agents each analyze one investigation and a second clustering stage surfaces systemic failure patterns across customer accounts. # Video Identity Observations — "Fighting AI with AI — Lawrence Jones, Incident" ## Explicit Self-Description (Creator Claims) - **Speaker identity**: "I'm Laurence. Uh I'm a founding engineer at a company called incident.io." - **Product scope claim**: "we build an incident response management platform… We page you when things go wrong and we help you run your incident." - **Customer claims**: "we're used by companies like Netflix, Etsy, Skyscanner, and actually probably a few of you in the room." - **Ambition claim**: "our goal is actually to fully automate kind of production investigations… we want to be the place that you turn to ask us questions about actually what's what's going on." - **Company status / hiring**: "we are hiring. Uh we we're in London. Uh we have just done a fairly big raise last year and we're looking to expand the team." - **Prior talk reference**: "I actually did a talk uh a year ago um at LDX about becoming AI engineers." ## Intended Audience - **AI engineers at an AI engineering conference**: "given that we're an AI engineering uh conference, are familiar with. So, things like prompts, evals, scorecards, traces, datasets, backtests." - **Builders of complex AI systems**: "for any of you in the room who are building kind of complicated AI systems um and you're finding it really hard to understand them or debug them or evolve them…" - **Practitioner-level assumed knowledge**: the talk assumes familiarity with evals, coding agents, Claude Code, and Codex without introductory explanation. ## Claimed Scope of the Talk - Topic: "how we use AI to manage the complexity of the AI products that we build um at incident.io." - Three stated areas: 1. "how you can use AI to help you manage and curate your eval datasets" 2. "starting to translate the UIs that we built to try and debug them um into downloadable file systems" 3. "how you can build kind of repeatable analysis pipelines using AI agents" ## Observable Presentation Patterns - **Educational, pattern-sharing format**: presents generalized takeaways ("file systems are exceptionally good agent context," "anytime you are performing complex analysis, look at creating an AI run book for it instead"). - **Internal-tool walkthrough style**: demonstrates concrete internal artifacts — an "eval tool" CLI, a "scrapbook" repo, YAML eval files, runbooks — rather than product marketing. - **Recruitment framing at close**: ends with a hiring pitch tied to the technical content. - **Engineering-culture signals**: "we do everything in go incident.io, including all of the AI work that we do"; "this is how we prove when we make a change to a prompt before we ever go and merge it." - **Quantified claims about internal performance**: "86% accurate RCA on our account" from daily backtests; "we're running thousands of of investigations across hundreds of our customer accounts." ## Claims vs. Demonstrated Evidence (Distinction) - **Claimed**: full automation of production investigations — this is stated as a *goal*, not demonstrated: "our goal is actually to fully automate kind of production investigations." - **Claimed**: customer list (Netflix, Etsy, Skyscanner) — asserted, not verifiable from the material. - **Demonstrated**: engineering practices (eval workflows, file-system debug downloads, agent analysis pipelines) are shown via described artifacts and worked examples, e.g., the pirate-speak prompt example and the "steal an eval from production" button. - **Claimed effectiveness**: "It will save you literally days or maybe weeks of your life" — presenter assertion without supporting data in the material. # Delivery Observations — "Fighting AI with AI" (Lawrence Jones, incident.io) ## Tone & Overall Style - **Conversational, casual engineer-to-engineer tone** throughout, marked by heavy filler usage: "um," "uh," and "like" appear constantly. - > "Um hi everyone. Uh so I'm here to talk today about..." - **Self-positioning as a practitioner sharing lessons**, not a polished keynote: "to share with you some of kind of the tips and tricks in the internal tools that we use." - Frequent **direct audience address** and assumptions of shared experience: "I'm assuming that several people in the room have kind of come across these themselves," and "for any of you in the room who are building kind of complicated AI systems." ## Vocabulary & Recurring Phrases - **Repeated intensifiers**, especially "really": - "really quite complicated," "really really fiddly," "really really hard to work with," "really really useful report," "really really large," "works really quite well." - **Hedging qualifiers**: "kind of" and "like" used as discourse softeners many dozens of times — e.g., "a kind of contrived prompt," "kind of like a unit test for your prompts." - **Recurring technical vocabulary** used consistently: *evals, prompts, runbook, red green cycle, back test, traces, file systems, coding agents, Claude Code, Code X, agents, RCA*. - **Framing devices** that recur: - "So yeah" as a section pivot: "So yeah, this is something that has turned the way that we debug our application into...", "So yeah, that's it." - "Um but the problem with this is like..." as a repeated problem-statement transition: - "Um but the problem with this is like this solves one problem..." - "But we now have another problem, right?" ## Teaching Habits & Structure - **Problem → solution → new problem** chain structure. Each solution introduced is immediately qualified by its limitation: - Evals work "really quite well, but it does have some problems" → eval tool CLI solves that → "but the problem with this is like this solves one problem" → file-system downloads → "But we now have another problem, right?" - **Rhetorical questions to the audience** to set up motivation: - "how would you actually figure out how to tell me if this was a good or bad report?" - "So, how on earth do we scalably understand how this system is performing?" - **Analogies to familiar concepts**: "evals for me are AI unit tests," "You want a unit test suite to be reasonably understandable." - **Walkthrough of contrived examples before real ones**: a "silly" pirate-speak prompt is demoed first ("this is kind of a contrived prompt... really simple, but silly"), then a "real prompt for us." - **Explicit signposting** of the agenda: "in this talk, I'm going to talk about how you can use AI to help you manage and curate your eval datasets... And then I'm going to talk about..." - **Step-by-step narration of agent workflows**, often in second person: "you sit there in the session and you go, 'Hey, like have a look at this...'" - **Concrete numbers used for scale**: "hundreds, if not thousands, of prompts," "maybe 25 agents in parallel," "86% accurate RCA on our account," "an hour or so," "thousands of investigations across hundreds of our customer accounts." ## Communication Patterns - **Simulated dialogue with coding agents**, quoting typical prompts: "Hey, I've got a problem here. Can you look at this prompt?" and "I really want it to do this instead. Like what part of the system would you change?" - **Named-nerd specificity** about tools and stack: incident.io does "everything in go," references "LDX," "Claude Code," "Code X," "MCP," "Cloud Code" (apparent verbal slips for "Claude Code" — [493], [714], [748]). - **Declarative takeaway summaries** at section ends: "file systems are exceptionally good agent context," "It will save you literally days or maybe weeks of your life." - **Closing with a hiring pitch**: "we are hiring... if any of this work looks interesting to you... just get in contact and let me know. I'd love to chat." ## Pattern Confidence - **High confidence**: filler-heavy casual delivery, "really/kind of/like" hedges, problem→solution→problem chaining, rhetorical questions, unit-test analogy, "So yeah" pivots — all heavily repeated in the transcript. - **Moderate/uncertain**: Tool-name slips ("Cloud Code," "core code," "Cloud Cove") appear a few times and likely reflect transcription noise rather than the speaker's habit; the exact spoken form is uncertain from transcript alone. # Fighting AI with AI — Lawrence Jones, Incident.io ## Speaker - **Lawrence Jones**, founding engineer at **incident.io** (identified from transcript introduction). ## Background & Product Context - incident.io builds an **incident response management platform** used by Netflix, Etsy, Skyscanner, and others; they page teams and support incident communication. - Jones states the company's goal is to **fully automate production investigations** — from major incidents to simple tickets — as the place users ask "what's going on." - The system has been built for **~1.5–2 years** and is, in his words, "on the edge of what you can do" with current AI technology. - The systems are now too complex for humans to debug tractably: "you can't as a human really tractably dig into how these things are performing." - An investigation at incident start runs **hundreds of telemetry queries** (logs, metrics, traces, historical incident data), cross-referenced with the codebase, to suggest a likely root cause and fix. - A single incident takes a human **about an hour** to fully understand well enough to judge whether an investigation report is correct. - Behind one investigation are "hundreds, if not thousands, of prompts" — motivating using AI to understand AI systems at scale across many customer accounts. ## Framing - Jones references a prior talk at **LDX** on "becoming AI engineers," covering prompts, evals, scorecards, traces, datasets, backtests. - This talk assumes those constructs and addresses: using AI to **manage/curate eval datasets**, translating debug UIs into **downloadable file systems**, and building **repeatable AI analysis pipelines**. ## Evals as AI Unit Tests - Definition given: an eval takes a prompt, runs it on input data, and applies grading criteria to pass/fail. - Eval YAML files live next to Go prompts in the codebase; all incident.io AI work is in **Go**. - Example (deliberately contrived): a prompt translating messages into **pirate speak**, graded on (1) looking like pirate speak and (2) preserving meaning. ### Stated Problems - Evals are "really, really fiddly" — realistic test data is hard to construct. - Production evals may require **almost an entire incident report** as input; these become unmaintainable. - A "steal an eval from production" button exists, but production evals are poor unit tests — "you don't want to have like 2 MB of all the YAML." - Large YAML files **broke coding agents**, which hit context limits when loading them. ### Solution: eval tool CLI - A small CLI ("eval tool") lets agents list, edit, replace, and add test cases. - A **runbook/skill for coding agents** drives a workflow: create an eval proving the failure → modify the prompt → verify the new eval passes → check the change hasn't broken other evals → consolidate the prompt to prevent unbounded growth. - Jones shows a real example (prompt converting human queries into log queries for a "low-key" system) where the agent adds an eval, repeats runs to check the pass rate, and concludes. ### Remaining Limitation (preserved as stated) - This solves the problem *only when you know which prompt to change*. Production AI systems are many prompts: the incident chatbot graph has roughly **10 agents** and ~50 prompts/tools — "actually bigger than this I couldn't fit it on the screen." A bad customer interaction doesn't reveal *which* component is at fault. - The investigations system is worse: each step expands into "possibly hundreds of different prompts and hundreds of different tool calls," and a subtle error anywhere can't be easily traced to its origin, even if it produces a totally wrong RCA. ## Downloadable File Systems - Existing debug UIs helped humans but not agents. - Jones attributes the insight to Anthropic's Claude Code: agents work "fantastically" with file systems and standard tools. - incident.io now lets each AI system's UI content be **downloaded as a self-documenting file system**, dropped into sandboxed Claude Code. - Workflow: report a bad experience → download the interaction → ask Claude Code "what went wrong / what would you change?" → it traverses the prompt/tool hierarchy and identifies the modification point → with codebase access, make the change → prove it via the eval red-green runbook. - Traces translate "exactly" from UI to text files; "there is really not much of a limit as to what you can put into ASCII." ## Backtests & Analysis Pipeline (Scrapbook) - Daily **backtests** run batches of investigations across their own and customer accounts, yielding a rolled-up number (e.g., "86% accurate RCA") — but Jones notes this doesn't explain *why* the number moved up or down. - Solution: download all investigations into a file system fed to an **analysis pipeline run with Claude Code**, using **markdown playbooks** for repeatability. - A repo called **Scrapbook** contains this structured flow: - **Parallelize**: ~25 sub-agents analyze individual investigations. - **Cohort clustering**: group failures into meta-patterns. - Output: a report on why the system performs well/badly per account and what to fix. - Recommended pipeline properties: - Leverage sub-agents for per-entity parallel analysis. - Store analysis in files inside downloads for incremental, resumable analysis. - Combine analysis with the powering codebase so problems point to specific code and suggested changes. - End by having the coding agent (Claude Code, Codex) make the change and confirm via the eval red-green process. - Jones shows a resulting PR: backtest flagged failing investigations, he identified the problem, discussed a feature change, deployed, and tested in production. ## Key Takeaways (per Jones) - These patterns **generalize** to anyone building complicated AI systems. - **Prioritize debugging tools that work well with coding agents.** - **File systems are exceptionally good agent context** — better than MCP or human-use agents, because bulk download lets agents grep through everything. - For any complex analysis, **write an AI runbook** — "It will save you literally days or maybe weeks of your life." - incident.io is **hiring** in London after a recent raise. # Video Attribution Observation Record ## Video Identification - **Title:** Fighting AI with AI — Lawrence Jones, Incident - **Video ID:** `01a15ca9-495a-7147-a12a-8ccc61924c57` - **Format:** Conference talk (single presenter) with audience applause at the end ## Speaker Inventory | Speaker | Role | Evidence | |---|---|---| | Laurence (spelled "Lawrence" in title, "Laurence" in transcript) | **Creator / Presenter** — introduces himself as **Founding Engineer at incident.io** | Self-introduction: "I'm Laurence. I'm a founding engineer at a company called incident.io." | - **No guests, narrators, or other identified speakers** appear in the transcript. - Audience presence is implied (applause, "a few of you in the room") but no audience members are individually identified. ## Referenced Sources & Third-Party Claims - **Customer claims (speaker's own assertion, unverified):** incident.io is "used by companies like Netflix, Etsy, Skyscanner." - **Prior talk referenced:** The speaker cites his own talk "a year ago at LDX about becoming AI engineers." - **Third-party tools/products named by the speaker:** - **Claude Code / "Cloud Code"** (Anthropic) — credited to Anthropic: "Anthropic stumbled on this with Claude Code" - **Codex / "Code X"** - Internal tools: **eval tool** (CLI), **scrapbook** (analysis repo) - **Technical context stated by speaker:** incident.io builds an **AI SRE product** and **incident response management platform**, written in **Go**. ## Promotional Segments — Flagged > **Hiring/promotion segment (closing remarks):** > "We are hiring. We're in London. We have just done a fairly big raise last year and we're looking to expand the team… get in contact and let me know." - The speaker also describes incident.io's customer base and product goals during the introduction, which serves partially as **company promotion**, though embedded in informational framing. ## Availability & Provenance Limits - **No curated metadata supplied** — only transcript provided; no channel, date, or speaker bio verification available. - **Comment availability:** Not indicated (no metadata on whether comments are open or disabled). - **Speaker name spelling discrepancy** in source: title says "Lawrence," transcript says "Laurence" — recorded here without resolving. - No timestamps available in the transcript; time positions of segments cannot be cited. \>\> Um hi everyone. Uh so I'm here to talk today about how we use AI to manage the complexity of the AI products that we build um at incident.io. Uh and to share with you some of kind of the tips and tricks in the internal tools that we use when we're building um our AI SRE product. Uh but first I guess like who am I? Um so I'm Laurence. Uh I'm a founding engineer at a company called incident.io. Uh so we build if you haven't heard of us, uh we build an incident response management platform. So we're used by companies like Netflix, Etsy, Skyscanner, and actually probably a few of you in the room. Um we page you when things go wrong and we help you run your incident. And as you're running your incident, we help you communicate with your customers. Um but like you might be thinking like where does AI actually come into this? Um and actually that like we don't just want to help people respond to these incidents. Um our goal is actually to fully automate kind of production investigations. So whether or not it's a big incident or if you just have some ticket and you wanted to look into production, uh we want to be the place that you turn to ask us questions about actually what's what's going on. Um now it turns out we've been building this for about a year and a half, 2 years now. Um and that's actually like a really big ask and the systems that we've had to build to try and support this have been really quite complicated um and have been kind of on the edge of what you can do uh with all of the AI technology that's out there. Um and they often pose a challenge for humans to debug them. Uh they are now complicated enough that you can't as a human really tractably dig into how these things are performing. Uh you need assistance to help you. Um so for example, this is a kind of uh one of the investigations that we would actually produce for you uh when you have an incident. Uh we will end up running uh right at the start of the incident this investigation which will go through hundreds of telemetry queries. It's going to look at your logs, your metrics, your traces, any historical incident data that we have, um and it's going to try and cross-reference this with your code base and go, "Hey, like, I'm pretty sure that the problem is this and you should probably do this to fix it." But, I want to pause here and go like, how would you, if you were building this system, how would you actually figure out how to tell me if this was a good or bad report? How do you know if it's right? How do you know if it's wrong? Um and there's a lot of things that you might do. Uh you might jump into the incident, you look at everything that happened, you might look at the postmortem if there was one that was written. Um but, all of this might actually take you a really long time to do. In fact, it normally takes you like an hour or so to get a real full understanding of this incident. Uh and it's only at that point that you could then look at this investigation and go, "I think it's right." or it gave me the information that was really, really useful. Um and as I said, behind this investigation is like hundreds, if not thousands, of prompts. So, how on earth do we scalably understand how this system is performing, especially across all of our customer accounts when they all have very different things going on? Um you end up with a lot of stuff, a lot of AI, and you've got to use AI to try and actually tractably get a handle on this. Um so, I actually did a talk uh a year ago um at LDX about becoming AI engineers, uh where I went through some of the core constructs that hopefully like a lot of you in the room, given that we're an AI engineering uh conference, are familiar with. So, things like prompts, evals, scorecards, traces, datasets, backtests. Um this talk is going to be about, if you assume that you have these constructs put together and you're building these complicated AI systems, um how can you use AI with the internal tools that you use to understand them to get a better handle of how your system is performing? So, yeah. Um in this talk, I'm going to talk about how you can use AI to help you manage and curate your eval datasets and making it easier for you to work with them, uh making it easier for coding agents to actually work with your eval tool. Um I'm going to talk about like probably what was the biggest unlock for us when we were building these systems, which was starting to translate the UIs that we built to try and debug them um into downloadable file systems, which has actually helped us massively using tools like Cloud Code and Code X to dig into how the system is performing. And then I'm going to talk about how you can build kind of repeatable analysis pipelines using AI agents to run through them. Um but first, evals. So, evals for me are AI unit tests. So, each eval takes a prompt and it goes, "Here is an input data." It runs the prompt, it gets the output, and then it has some grading criteria that says, "Does this eval pass? Does it fail?" And for us, evals live in YAML files right next to all to like our go prompts. So, we do everything in go incident.io, including all of the AI work that we do. And this is how we prove when we make a change to a prompt before we ever go and merge it that the prompt is actually going to do the thing that we want it to do. Um so, for us, this is what a prompt looks like. This is kind of a contrived prompt. So, I would hope no one actually has this in production anywhere. It takes a message and it tries translating it into pirate speak. So, really simple, but silly. But, what we do for evals is if this is the prompt, we would then define on the left some grading criteria for this prompt where we'll go, "There are two things that we care about. We care that the result actually looks like pirate speak, and we are going to care that the meaning is preserved between the input and what we actually produce as an output." So, this is actually what we're going to use to tell us if the eval passed or failed. And then we have the evals up on the top right, which is just in a YAML file where we go, "Here are three different test cases." And we'll run through them, and you can see the results of us actually running this on the bottom right. So, this works, and it works really quite well, but it does have some problems. And I'm assuming that several people in the room have kind of come across these themselves. So, first, um like evals are really, really fiddly. Setting up realistic test data for your evals, if you want to actually understand how this stuff is running, is is difficult to do. And especially in our environment, our production evals are including almost an entire incident. So, you can imagine a full incident report, and that's the only thing that can trigger the bad behavior. This is kind of hard for you to end up pulling them down and putting them in your eval test suite. Um they just become extremely unmaintainable very quickly. Uh now quite early on we created like this little button that we have that allows you to like steal an eval from production. So, if anything was going wrong inside of our like AI interactions, you could go in and you could just pull that down, put it in the code base, and you could run the eval against it. Um but the thing with this is like production evals aren't like great. If you think about evals as kind of like a unit test uh for your prompts, you want a unit test suite to be reasonably understandable. Like an ideal unit test is very focused and just says I expect it to do this thing. You don't want to have like 2 MB of all the YAML associated with it. It's just really really hard to work with. Um and what we found was as these YAML files for the evals were growing really really large, uh our coding agents weren't able to work with them. So, if you want to do a quick like read and like modify the eval suite, you'd be boosting that into the context, and you'd quickly hit your context limit, uh which is obviously a problem cuz then you can't work with it effectively. Um so, what we ended up doing was we ended up creating a small CLI tool that we call eval tool um that was designed to allow agents to leverage our eval suite files. Um so, it's just a small CLI that can go what test cases do you have in here? I want to edit one, I want to replace one, I want to add one. Um and it was by doing this that we allowed agents to work effectively with our eval tooling, and that's why we were able to create this runbook to the right, which is actually a runbook that's designed for um a coding agent to use. So, either a runbook or a skill, it depends on how you want to package it. Um but the cool thing about this is that now that agents can work with the evals, you can end up in a situation where you just ask your coding agent to go, "Hey, I've got a problem here. Can you look at this prompt? I want it to do these things." And the coding agent's going to turn up, and it will create an eval case where it proves that the thing is failed and then it will go modify the prompt so that the eval now passes and then it will go through this run book and one of the most important stages for us is checking at the end that the change that you've made to the prompt hasn't ended up breaking any of the other evals that you had in your test suite and we also have like a final pass that tries consolidating the prompt as well cuz if you end up doing this repeatedly you end up with a prompt that is massive and really really difficult to maintain so you kind of want every time you make an adjustment to try and simplify as well. Um so this is actually like like really well for us. Um and you can see here this is like me using it in core code where you can just point it at the eval and say hey have a look at the prompt. This is a real prompt for us which turns human queries into log queries for a low-key system and it ends up racing through and goes and adds a new eval it checks that it passes with a certain number of repeats and then it gets to the end and it's like yep I think I've added it kind of the pass rate is acceptable you can go ahead and get going. Um but the problem with this is like this solves one problem um and the problem that it solves is that if you know what the prompt is that you want to change you can now change it very reliably and that's very useful if you're working on these tools but one of the biggest problems that you have now is that if you're building these systems you'll know that they're not just one prompt anymore. Um in fact most of the production AI systems you will use on a daily basis are many many many prompts. Um and to illustrate this problem I've taken our chatbot. So this is the chatbot that you interact with inside of an incident and what I've done is I've created a graph of all of the different prompts tools agents and everything in the hierarchy that powers an interaction with our system. Um so you can actually see there's like 10 different agents there there's 50 I I don't even know there's it it's actually bigger than this I couldn't fit it on the screen. Um there's a lot of stuff so even if you think that you know even if you've got a bad interaction that came in from a customer you don't necessarily know which part of your system is actually the problem and which part to go change. Um so even if you have this eval red green cycle, you're going to struggle to know where to go to fix it. Um and this gets even worse for a system like our investigations. So, if you think about trying to run through this process to debug what's going on an incident, we have a ton of stuff that goes on uh inside that system and you can see all the steps on the left. Um and each one of those steps unpacks into the trace that you have on the right and like really it's not about the details here. Um it's more about each one of these green blocks actually expands into possibly hundreds of different prompts and hundreds of different tool calls. And at any point, if you make a slight subtle error, you can't then easily trace through the system where the error originated even if it ends up resulting in you having totally the wrong picture on what you think that the incident was and your RC is totally wrong. Um so like we built these UIs so that we could help humans look at them and they've been really good for humans to look at them, but I guess going back to what I was saying before, um we just don't have enough time to go through this stuff. So, the problems that we had was we have all these UI tools, but agents can't properly use them. So, how do we get to a place where like the agents can use the tools properly? Um and I think Anthropic stumbled on this with Claude Code where um they found kind of when they released Claude Code that these agents are fantastic at using file systems and just going through this data using standard tools. Um so we kind of thought like, can we just download all of the UI that we have as a file system? Um and that's kind of what we've done. So, now for each different AI systems, you're able to download all of the content as a file system and we drop that into a sandbox Claude Code uh at which point you can just point Claude Code at it and go, "Hey, I've got a problem here. It's behaved in the wrong way." It can see everything that went into all the prompts, it understands the structure cuz it's self-documenting. Um and then it can tell you because you have access to the code base as well, uh exactly where you should actually be making the modification to try and change it. And then you can lean on that red green cycle from before to try and modify prompt if you need to. Also, there's more stuff that you can put in this than you might think. There is really not much of a limit as to what you can put into ASCII. Um so like traces like this can get translated exactly from how you would present them in the UI to a text file which then the LLM can consume in a really nice way. So yeah, this is something that has turned the way that we debug our application into we hear that there's a bad experience. We end up downloading that interaction into a sandbox called code. You sit there in the session and you go, "Hey, like have a look at this. Tell me what you think has gone wrong. Like like what is your interpretation of the problem?" And then you go, "I really want it to do this instead. Like what part of the system would you change?" And then it will work its way through the hierarchy of all those tools and prompts that you just saw and it will be able to tell you where you should be making the modification. And then all from that session because you have access to the code base, you can just go, "Hey, can I make that change?" And then you can prove it using the Eva runbook that I mentioned before. So yeah, like we've implemented these file system packages for a load of different AI interactions for us now. So it's really easy for us to just drop this in Cloud Cove and just get going. But we now have another problem, right? Cuz whilst you can do this on an individual basis, we are running thousands of of investigations across hundreds of our customer accounts. And we're doing that daily cuz we need to know if this system is getting better or worse. So what you can see here is we have what we call a back test which is essentially a batch of investigations that we run on a daily basis against our account and against a load of our customer accounts as well. And eventually you just get this rolled up number which is like, "Oh, cool. 86% accurate RCA on our account." Which is which is great, but like this doesn't really tell you why the number went up and it doesn't tell you why it went down. And if you want to improve the system for someone, you're going to struggle. So what we've actually done is we've allowed ourselves to download kind of all of these investigations into a file system that we can then provide into an analysis pipeline that again is leveraged or is run using Claude code that can end up running a structured analysis like with markdown playbooks that help you run it repeatedly and reliably each time. Um so what that actually looks like is we created this repo this repo called scrapbook and inside of scrapbook we have this like very structured flow that explains exactly how a coding agent should go through all of the information that we've gone and downloaded, how it should understand these investigations and the process that it should go through to actually run them. Um now the key things that like I think are very important to these flows are you start and you actually parallelize out all of your agents. So you start maybe 25 agents in parallel and they can all individually build their analysis of an investigation and then you go into the next stage of the pipeline where you do some cohort clustering and you look at like meta points around like what are the same types of failure, how do we go wrong in different ways. Um and by clustering it together you end up with actually like a really really useful report that doesn't just tell you how this has gone wrong but it tells you why is your AI system performing well or badly on this customer account and actually like what should you do to try and fix it or improve the system. Um and like this is this is like something that we've done several times over for several of our systems now and I think it generalizes really well for anyone who's building this type of thing. Um so the points that make like a really good pipeline for this you should leverage sub agents to do that parallel per entity analysis. You should store all of your analysis in files inside these downloads so that you have like incremental analysis built as you run through it so that you can start and resume the analysis if you have need to. Um and then you want to combine this analysis with the code base that is actually powering the system. So that if it finds a problem it can look in the code base and go hey I think that this is the problem and this is the place and it can actually do some analysis to go I think that you should change it like this. Um and then at the end because you have this all loaded in your code session you can just go fix it and ask the coding agent Claude Code, Code X, whatever you use to actually go make the change and then you can use that eval red green process to actually confirm it works. Um and then like yeah, this is a PR that is created after you do something like that where because the back test showed a couple of investigations that were going wrong and I knew exactly what what the problem was, um I could have a chat with you about a feature that we might change in the system. Um and then we can deploy that and we can test that out in production and see how the thing goes. Um so yeah, that's it. Uh so um the key thing for me is like these patterns do generalize. So for any of you in the room who are building kind of complicated AI systems um and you're finding it really hard to understand them or debug them or evolve them, um you you really need to be using AI just as effectively in your internal tools to try and understand these systems and grow them um just as you are in the products that you're building yourself. Um so yeah, make sure that you prioritize any of the debugging tools that you have so that they work really really well with the coding agents that you're leveraging on a day-to-day basis. Um file systems are exceptionally good agent context. Like we could have put an MCP on top of this or use like human use uh agents. Uh it wouldn't have been half as effective as this ability to just download in bulk all of the information that you need so that the coding agent can grep through it and find the details. Um and then yeah, anytime you are performing complex analysis, look at creating an AI run book for it instead. Um it will save you literally days or maybe weeks of your life. Um and then one final point for me, um like we are hiring. Uh we we're in London. Um we have just done a fairly big raise last year and we're looking to expand the team so that we can build some of these systems. Um so if any of this work looks interesting to you and you're interested in being on like the edge of building some of this AI like AI SRE product, uh then just get in contact and let me know. I'd love to chat. All right, thank you. \>\> \[applause\] \>\> Mhm.