From Blind Spots to Merged PRs: Continuous Agentic…
Conference talk by May Walter (Co-Founder and CTO of Hud) presenting a case study of adding runtime intelligence to coding agents to enable continuous performance optimization in mature production codebases. Covers the pain points of unpredictable, hard-to-justify performance investigation, the technical approach of agents analyzing real production context to surface high-ROI fixes scored by complexity and impact, and improvements needed for reliable results. Reported outcomes, per the talk's claims, include surfacing and fixing N+1 queries and missing database indexes in the first week, measurable P90 latency improvements, and actionable reports delivered to tech leads before sprint planning.
# Channel Presentation Observations
## Explicit Self-Description
- The speaker identifies himself: **"I'm Mike, co-founder and CTO at Thundra."** He describes the company as building **"a runtime intelligence layer for coding agents that captures functional level context and deep forensic context… so that coding agents can help you fix what's going on with production."**
- Note: the video title credits **"May Walter, Hud"**, while the transcript self-identification says **"Mike, co-founder and CTO at Thundra"** — a discrepancy between title metadata and in-transcript claims.
## Intended Audience
- **Software engineers and engineering leaders** working with AI coding agents and production performance:
- "if you feel like that, I want you to know that you're not alone" — addressing teams struggling to trust agentic PRs.
- "I'm sure you guys also experienced that when something throws an exception and then the agent says…" — assumes audience familiarity with agent behavior.
- "hopefully you can take something out of it and apply it in your day-to-day."
- Audience is assumed technical: references to **ClickHouse, GitHub Actions, MCP, Slack, P90/P99 metrics, MongoDB indexes, SLO breaches** are used without basic explanation.
- Closing invitation to a peer community: "looking forward to hear about cool automations that you build on your own."
## Claimed Scope of the Talk
- Structured explicitly by the speaker:
- "I'm going to start with why we even wanted to do this. Then we're going to go over the tech and the process… and then share some gotchas and takeaways along the way."
- Framed as a **conference-style presentation** (greeting, closing Q&A offer: "if there are any questions… I'm available").
## Observable Presentation Patterns
- **Vendor-neutral, tool-agnostic framing**: "we wanted an infrastructure… vendor neutral in terms of compute… the harness and… the model," while naming specific choices ("GitHub agent workflows," "Cloud Code… that was our specific choice," "it could have been Cursor or Copilot").
- **Balanced candor about limitations**: shares failure modes — "the plausible unverified," "the lazy fix," "it didn't work out of the box if you were wondering," and the failed assumption that "everyone will fix everything and the world will be amazing."
- **Human-in-the-loop emphasis**: repeatedly stresses human review gates and trust thresholds ("crossing towards the 80-90% trust"), suggesting the channel/venue values pragmatic adoption guidance over hype.
- **Cites external data**: references "Google just published their Dora metrics for 2026" to ground claims about AI's impact on delivery instability.
- **Educational takeaway structure**: ends with "four things that I learned that could be relevant for you," reinforcing a lessons-learned presentation format.
## Demonstrated Identity Signals
- The talk doubles as a **product-adjacent narrative**: the described workflow is built on Thundra's own technology ("the HUD, the heads-up display," "the HUD query language," "prod-to-code"), indicating the presentation showcases the company's platform while offering transferable practices.
- Tone is informal, first-person, anecdote-driven (the "Jenny" and "only Dave" opening story), positioning the speaker as a practitioner sharing experience rather than pure marketing.
# Delivery Observation Report
## Speaker Identification Note
The transcript shows a **possible discrepancy**: the title lists "May Walter," but the speaker introduces himself as:
> "So, hi. I'm Mike, co-founder and CTO at Thundra."
(This is a factual observation from the transcript; the reason for the mismatch is not observable.)
---
## Tone & Register
- **Conversational, peer-to-peer**: casual phrasing like "the dream is very nice, but but the devil's in the details" and "I'm sure you guys also experienced that."
- **Empathetic framing toward the audience**: "if you feel like that, I want you to know that you're not alone."
- **Self-deprecating humor and asides**: "for me personally, lack of sleep is another symptom of that."
## Narrative & Structural Habits
- **Opens with a relatable mini-drama** using invented characters before introducing himself:
> "And then a few weeks later, Jenny would say, 'No, no, no, but it's actually way too slow now.'"
> "the answer is only Dave. He's the only one who kind of knows that code"
- **Explicit roadmap statements**: "I'm going to start with why we even wanted to do this. Then we're going to go over the tech and the process... and then share some gotchas and takeaways along the way."
- **Numbered summary close**: "So, four things that I learned that could be relevant for you. One is..."
- **Uses vivid metaphors/labels**: "leaky bucket," "black box," "low-hanging fruits," "the plausible unverified," "prod-to-code," "rain of 80 pull requests."
## Recurring Phrases & Verbal Patterns
- **"Kind of" as a frequent hedge** — appears repeatedly: "kind of leave it as is," "who kind of knows that code," "all look kind of good to us," "Kind of like that performance sprint," "it's kind of a way to explain."
- **"So" as a discourse marker** to begin or reset sections: "So, we're surrounded by...", "So, this is basically how it looks like," "So, what we actually do is..."
- **Repeated self-corrections / false starts** (observable in transcript): "but but the devil's," "it it just didn't work," "in a in a while," "and and require attention."
- **Conversational tags**: "you know," "I guess," "um," "cuz."
## Teaching Habits
- **Concrete hypothetical examples with specific numbers**: "usually taking around 200 ms, but every once in a while it takes like 45 seconds"; "running 7,000 times a week."
- **Dialogue-styled illustration**: quotes hypothetical agent speech — "Well, maybe we can just catch that exception and then everything will be fine."
- **Contrast framing** to build points: "It's not hey, I have this idea of something that you can do. It's here's something that works."; "agentic engineering is not like coding with an agent."
- **Frames every solution as having a "how it actually went" reality check**: "And of course it didn't work out of the box if you were wondering."
## Confidence & Uncertainty Markers
- **Hedges absolute claims**: "I'm not saying it's never going to work"; "they're not necessarily the absolute right thing because it doesn't necessarily exist."
- **Honest acknowledgment of difficulty**: "It's legitimately hard to prioritize something where you're not sure exactly what you're going to get out of it."
## Stable vs. Isolated Features
- **Stable patterns**: "kind of" hedging, "So" as a reset marker, contrast framing, concrete numeric examples, numbered takeaways — each appears multiple times.
- **Isolated moments**: the "leaky bucket" metaphor and named characters (Jenny, Dave) appear only in framing passages; insufficient repetition to call them signature devices.
## Uncertainty Statement
Transcript-only evidence means **paralinguistic delivery** (pacing, gestures, slide interaction) is not observable; there is one "[clears throat]" marker at [310]. Observations above are limited to lexical and structural features of the transcript.
# Observation Record: "From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization"
**Speaker:** Mike (self-identified in transcript as co-founder and CTO at Thundra). Note: video title metadata lists "May Walter, Hud" as presenter; the transcript self-identification says "Mike." This tension is preserved unresolved.
## Opening Problem Framing
- Mike describes a common team dynamic: product raises a performance concern, engineering defers investigation, and eventually the issue escalates. He characterizes the investigation cost as unpredictable:
> "We would know how much time it takes to do a specific optimization, but we never know how long it's going to take to investigate it."
- He claims teams rarely proactively look for performance/stability improvements even though they are found "every time we look."
## Company Context
- Thundra is building a **runtime intelligence layer for coding agents**, capturing function-level and forensic context so agents can help fix production issues. As part of this, they built an agentic workflow for continuous performance optimization of production applications.
## Motivating Data
- Mike cites **Google's Dora metrics for 2026** (as published, per his statement):
- Biggest impact of AI adoption: **individual effectiveness**
- Second: **software delivery instability**
- **Throughput not impacted as much as expected**
- His summary: teams feel faster individually, throughput stays the same, and software breaks more often. He qualifies this with "which is not exactly what we were hoping for... just yet," and suggests agents could help "fix faster" to capture the gains.
## The Leaky Bucket Problem
- Issues get ignored until they degrade to crisis, get fixed in emergency mode, then return to being ignored — "a leaky bucket by definition."
- Root cause he identifies: the **research phase is a black box** ("It could take an hour or weeks"), making prioritization hard.
- Proposed solution: automate the investigation — weekly runs with real production context, scoring and flagging **high-ROI opportunities**, "kind of like that performance sprint that you run every few months, just automate it."
## Infrastructure Requirements
The stated requirements for the agentic workflow infrastructure:
- **Vendor neutral** — compute, harness, and model
- **Secure** — tool calls, permissions, authentication
- **Trigger system** — scheduled runs (weekly) or webhooks (e.g., SLO breach)
- **Easy to maintain/update** — because "expectations go up" over time and a feedback loop is needed for trust
**Choice made:** GitHub agent workflows, with (their starting setup) Claude Code, runtime intelligence captured via MCP, reports sent to Slack. He explicitly notes these were choices, not endorsements — "It could have been Cursor or Copilot."
## The Process Loop
1. Take production context (traces, queries, latencies), analyze with the agent
2. Score and flag what matters (an optimization running "every 3 weeks" or saving "20 milliseconds... doesn't matter")
3. **Agent fixes it, reruns tests, verifies impact** on the specific flow
4. Human reviews as a gate — "the loop is actually closed"
- Distinction he draws: not "I have this idea" but "here's something that works" tied to business flows running e.g. "7,000 times a week."
## Hurdles Encountered
- **"Plausible unverified" fixes** — suggestions that sound right but fail verification; the goal was grounding in actual production behavior.
- **Complex queries** — ClickHouse differs from classic SQL patterns.
- **The "lazy fix"** — e.g., catching an exception instead of understanding why it was thrown.
- Response: methodology and "building the playbook of how a senior engineer would do that."
## Context Problems
> "There are only two problems with context. You either have too much of it or too little of it."
- He claims both often coexist: lots of **low-signal data** hard to reason over, plus missing logs/traces/metrics that leave room for assumptions.
- Metrics (service level, CPU/memory, endpoint P90s) are **not connected to function level**, while agents reason over code — a language mismatch that hurts accuracy.
## Solution: "Prod-to-Code"
- Explain production behavior at the **function and file level** where agents reason, connected to the triggering endpoint/consumer/cron.
- Enables questions like "this endpoint that sometimes takes 7 seconds, where is the time spent?"
- Forensic capture only when needed (e.g., requests above P99/threshold).
- **HUD** (heads-up display) overlays data on code.
- Layers built: **HUD query language** (ClickHouse queries) → **skills** (reduce eval variance, e.g., HTTP 500 root-cause, memory spike baseline diffing) → **automations** (auto-fixing, dead code detection, performance improvement).
- Performance patterns searched: artificial delays (timeouts, sleeps), N+1 queries, missing indexes, sequential scans — "much easier to find when you actually look for them." He notes a 20-year-old codebase with hundreds of contributors will have many.
- Qualification on static analysis: "I'm not saying it's never going to work," but automation needs robustness to trust over time.
## The Human Adoption Problem
- Opening many PRs automatically failed: "no one wants to wake up for... a rain of 80 pull requests."
- Mitigations:
- **Prioritization**: hot path frequency, business criticality (payments, sign-up), risk (migrations require more review time)
- Seeking **highest ROI**, not highest impact, since investigation is automated
- Started with **one PR at a time** to build appetite and habit
- A **human-friendly report** with a short explanation, e.g., endpoint usually ~200ms but sometimes 45 seconds due to using `distinct` instead of a Mongo search index — user can create a ticket, PR, or ignore.
- Rationale: "we're still living in this hybrid world where humans are in the loop" — confirming both that the fix is good and that the issue is worth fixing.
## Stated Takeaways
1. **Define what matters** — scoring and guardrails make it reliable; fixes are cheaper now "but still not free."
2. **Automating a phase that didn't happen** — no engineer actually did weekly low-hanging-fruit reviews; agents surface opportunities that "would probably never" otherwise be done.
3. **Context over cleverness "works almost every time"** — with right context and skills, agents get more useful; he believes models are "good enough already" for most cases, but steering remains our job. He adds a caveat: "They're not necessarily the absolute right thing because it doesn't necessarily exist."
4. **Agentic engineering ≠ coding with an agent** — 80% reliability is fine interactively; autonomous automation needs **80–90%+ trust** so it doesn't "hand us a bunch of things that we would either waste time on reviewing or just not be able to trust."
- He identifies the hardest part: reaching confidence that the issue is worth fixing and the fix is verified in runtime before handing to a human.
# Speaker Attribution Record
## Video Metadata
- **Video ID:** `01a15ca8-7763-7b9b-8dee-556decf25feb`
- **Title:** From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization - May Walter, Hud
## Speaker Discrepancy
- **Title attribution:** "May Walter, Hud"
- **Self-identified speaker in transcript:** "I'm Mike, co-founder and CTO at Thundra"
- **Note:** The transcript's self-identification conflicts with the title's credited speaker. Uncertainty is explicit; no resolution is possible from the source material.
## Speakers and Roles
| Speaker | Role | Evidence |
|---|---|---|
| Mike (per transcript) | **Creator / Presenter** — co-founder and CTO at Thundra | Self-introduction; delivers entire talk |
| "Jenny", "Dave", "someone from product/engineering" | **Hypothetical/illustrative figures** in the presenter's opening anecdote — not actual participants | Framed as a scenario, not real speakers |
No guests, narrators, or other identified roles are present.
## Referenced Sources and Claims
- **Google Dora metrics for 2026** — cited for claims that AI adoption's biggest engineering impacts are individual effectiveness and software delivery instability, with throughput less affected than expected. Cited by the presenter; no document link provided.
- **Tools/products referenced (presenter's own stack):**
- GitHub agent workflows / GitHub Actions (weekly scheduled runs)
- "Cloud Code" (transcript's wording for chosen agent)
- ClickHouse (columnar database)
- MCP (runtime intelligence capture)
- Slack (report delivery); Cursor, Copilot, Teams mentioned as alternatives
- Thundra's "runtime intelligence layer," "HUD query language," skills, and automations
## Promotional Segments
- **Company promotion:** The talk promotes Thundra, where the presenter is co-founder and CTO ("We're building a runtime intelligence layer for coding agents..."). Product capabilities (prod-to-code context, HUD, skills, automations, weekly performance reports) are presented as the company's own work.
## Third-Party Claims
- Google Dora metrics 2026 findings (as summarized above) are the only third-party claims; attributed by the presenter to Google.
## Availability Limits
- **Comment availability:** Not stated in transcript or metadata — unknown.
- **Curated metadata:** Speaker identity beyond the title is absent; the Mike/"May Walter" conflict remains unresolved.
- No timestamps, links, or external documents were provided.
Hi everyone. I don't know if you're familiar with what I'm about to show. But remember when someone from the product is like saying that some page is too slow and maybe we can optimize it.
And then someone from engineering would say, "Yeah, probably what we'd have to dig in to find out." So, we just kind of leave it as is. And then a few weeks later, Jenny would say, "No, no, no, but it's actually way too slow now. We have to prioritize it. How long is it going to take?" And we would say something like, "Well, somewhere between an hour and a week. We'd have to look at it to find out." And then when she asked who can even take it, the answer is only Dave. He's the only one who kind of knows that code and everyone else who wrote it left a decade ago. And amazingly, this still happens in teams all the time because we would know how much time it takes to do a specific optimization, but we never know how long it's going to take to investigate it and to find it. And miraculously, every time we look, we we actually find things that can be done around performance and stability, but we never stop to proactively look for them because it doesn't make sense. So, hi. I'm Mike, co-founder and CTO at Thundra. We're building a runtime intelligence layer for coding agents that captures functional level context and deep forensic context on things that matter so that coding agents can help you fix what's going on with production. And as a part of that, we built an agentic workflow that helps with continuously optimizing performance of production applications. And I'm going to share a bit more about what we built, what were the challenges along the way, and hopefully you can take something out of it and apply it in your day-to-day. So, we're surrounded by a lot of agentic PRs that all look kind of good to us. And one agent wrote them and the other one says, "Yeah, I I went over it and it looks fine." And we still feel that urge to to verify before we just merge it to production.
And then when it doesn't work, we end up asking ourselves, "How the hell did we get to this place?" And if you feel like that, I want you to know that you're not alone. So, Google
just published their Dora metrics for 2026 and we can see that the biggest impact of AI adoption on engineering is individual effectiveness or that feeling of, "Oh my god, I'm so fast. I can do everything in the world." And for me personally, lack of sleep is another symptom of that. But the second one is software delivery instability. And the throughput is actually not impacted as much as we expected. So, we basically feel more effective, we're more effective individually, but as a team, our throughput is kind of the same and our software breaks more often, which is not exactly what we were hoping for with this AI revolution just yet. But maybe there's an agent for that and if we can build faster with AI and we can fix faster with AI, then we can get those gains that we were talking about. So, I'm going to start with why we even wanted to do this. Then we're going to go over the tech and the process, which I think is also really important. And then share some gotchas and takeaways along the way. So, that leaks kind of faster than we can bail. We have these issues that we ignore because they're not important enough then they degrade to the point where they are important enough and then we reach this crisis mode where we are all hands-on-deck. We fix it in emergency mode and then we go straight back to ignoring, which is a leaky bucket by definition. And that mostly happens because the research phase is a black box. It could take an hour or weeks and we have to pay that debt and to make sure that we spend time and engine engineering time on it in order to even know what can be done about it. And that's hard. It's legitimately hard to prioritize something where you're not sure exactly what you're going to get out of it. But what if we can automate that investigation? So, we can basically run on a weekly basis with real production context, analyze the sweet spot in the and flag the high ROI opportunities in a way that's scored. So, it runs automatically without us having to stop and do something about it. It has the production context in mind and it can give a high ROI scored
performance opportunities of the things that are easy and impactful. Kind of like that performance sprint that you run every few months, just automate it. Now, let's talk about how we can
actually do it cuz the dream is very nice, but but the devil's in the details. So, first of all, we wanted an infrastructure for the agentic workflows that would be vendor neutral in terms of compute and where it runs, in terms of the harness and in also in terms of the model. Things are changing all the time. We wouldn't want to constrain ourselves to a specific vendor, specific model or anything like that. And especially like with hot, we want our customers to \[clears throat\] be able to use whatever agents they wish. We also wanted it to be secure in terms of the tool calls, permissions, and authentication. We want some trigger system, whether those are scheduled runs like the weekly run or a set of webhooks. For example, if we see some SLO breach, we would want to investigate it. And we really wanted it to be easy to maintain and update over time. I think one of the biggest learnings we've had with agentic workflows is that even if they work out of the box or we get to a point where we're happy about them. In time, we evolve and our expectations go up. So, just being able to maintain and update those logics and build that feedback loop was very important for us so that it's reliable and the people actually trust the outcome. Specifically, we chose to work with GitHub agent workflows for that and we can choose whatever agent we want to work with and build the workflows on top of that, but there are many other great tools that could be used for that. So, this is basically how it looks like. You can see that there's a description of the job and the goal and the analysis and then we basically go over the GitHub repository and generate that weekly deep insight report analyzing production data and finding those low-hanging high-ROI opportunities. So, for this specific setup, GitHub
actions runs weekly and it uses Cloud Code, that was our specific choice, and then captures the runtime intelligence overhead via MCP so that we can
basically look at the different endpoints, connect to the function level of overhead is there, and send a report to Slack. Of course, that could have been Teams or an email or anything like that. It could have been Cursor or Copilot. It's just the setup that we started with um to make sure that we have something that runs without us in the loop and sends that report to somewhere we actually live in, which is Slack. So, we want to take the production context, the traces, the queries, the latencies, and analyze them with the agent, score and flag which opportunities matter because if you can optimize something, but it runs every 3 weeks or you can like reduce 20 milliseconds, then it doesn't matter. And then the most important part here is the different the evidence. So, the agent actually fixes it reruns the tests and sees the impact that it had on that specific flow that was optimized. So that the human gets something after we already detected it. We understood the root cause. We understood why it matters to the business and we verified that the fix actually impacted that time. And then a human reviews that. And the loop is actually closed. So it's not hey, I have this idea of something that you can do. It's here's something that works. And we believe would make an impact on these specific business flows in production that are running 7,000 times a week. And then the human gets in the loop as a review gate. And of course it didn't work out of the box if you were wondering. So the first
hurdle we had along the way is what we call the plausible unverified. So the agent would suggest a fix. It sounds right. It would look fairly real and then after we verified
it it it just didn't work. And it's it's right that the agent suggested something that could theoretically cause that flow down. But what we wanted is to ground it on what's actually happening in production. Second part was complex queries. Um we specifically use ClickHouse. It's just an amazing columnar database but it's also slightly different than the classic SQL patterns. Now I'll talk about that in a bit. And the third one is the lazy fix. I'm sure you guys also experienced that when something throws an exception and then the agent says, "Well, maybe we can just catch that exception and then everything will be fine." But what we really want is to understand why that exception was even thrown in the first place or why are the results lagging. So those were things that we found that methodology could be very, very impactful with. It's not just the understanding of the data and how to connect it. It's also being quite thorough on what we want to do in that process and sort of building the playbook of how a senior engineer would do that. And of course, we need the context to be right. So, the problem with context there are also There are only two problems with context. You either have too much of it or you have too little of it. And what
we found around runtime context, especially from production, is that more often than not, you actually have these two problems together because on one hand, you have a lot of low signal data
that is hard for the agent to reason over. And on the other, you might not have all the logs and traces and metrics that you need in order to investigate that issue, which would still leave some room for assumptions and theories of that are not necessarily what our users are experiencing in production. And also, when we talk about metrics like service level, CPU and memory, or endpoints in the P90s, they are not connected to the function level. So, our coding agents reason over code and they look at these metrics and there are some relations between them, but they don't exactly speak the same language. So, when we ask questions about taking time and what can I do about it, we're often finding that there are some gaps there and again, accuracy could drop from that. So, what we did is what we call prod-to-code, which is to be able to explain what's going on in production in the same level that agents reason over because the agent's context lies on a function and file level, not on an service and endpoint level. So, basically, our context is running on a function level, and it is also connected to the endpoint or event consumer or cron job that ended up starting this task. So, you can ask a question like, "Hey, this endpoint that sometimes takes 7 seconds,
where is the time spent, and what can be done about it?" Whether that's a function or an outbound call to a database, an LLM, or another microservice. And
with that, you basically have the complete function level context for every single function, sort of like this service map, but on a function level of the different invocations, where they come from, how often they run, and the difference the context only when it's needed, only when we see requests that are taking longer than the P99 or some threshold that we can define. Then, we will capture those forensic evidence so that we can say, "Hey, here is an example of a request that took longer. Let's find out why." And we also have the ability to see that on top of the code, which is the HUD, the heads-up display, but but I guess it's kind of a way to explain how that data set is actually structured in a way that is much more comprehensive for a coding engineer to read and over. And then, we talked about the complex queries and what to do with them. So, the basic layer gave us the HUD query language, which are basically ClickHouse queries over that structure of functions and endpoints and forensics. On top of that, we also added a set of skills. We found that sometimes just querying the data is enough, but being able to get to the right query and to ask it again and again really created a lot of variance in our eval, and the skills actually help work with that data so that agents can use it. So for example, if we're talking about a 500, an HTTP 500, we would want to understand where that error came from. If we're talking about a memory spike, we want to understand what was running on those specific pods at that specific time where memory was higher and compare it to a baseline so that we see the diff. And all of these things were extremely helpful to be able to be a bit more methodological around how that works and to create more consistent results. And on top of that, there are a set of automations. So if we have the data, the query language, a set of skills, we can build automations on top of them like auto fixing issues as they
arise or detecting dead code that isn't even running for the last 60 days and eliminating it or this automated
performance improve- improvements automation that are we are talking about right now. So for performance, one example of that is to be looking for artificial delays like timeouts and sleeps and plus one queries, missing indexes, sequential assings and so on so forth. These are specific things that are much easier to find when you actually look for them. And in a code base that's 20 years old and has hundreds of contributors, it makes sense that you'll find quite a lot of those. And removing them is fairly easy and impactful. And then when you're asking something like, "Why are my endpoints taking long?" you can actually find the specific reasons and not just guess a bunch of static code analysis which could get you some result. I'm not saying it's never going to work. But, when you're talking about an automation, we have to think about how to build something that is robust enough for us to trust over time. And then we said, "Okay, now that our evas are looking good, we run weekly, we find real slow endpoints from production that are invoked. We score them, we test them, we verify them. Maybe we can just open pull requests and everyone will fix everything and the world will be amazing." Well, that's not exactly how that worked because people are still people and no one wants to wake up for, you know, a rain of 80 pull requests, as small as small as they can be. That's just not how people operate. And no one has time for that. We're too busy building other things and it's fine. So, what we actually do is we use that priority to make sure that we only flag the ones that matter and we actually started with one at a time to create that appetite and that habit. So, we look at whether this is a hot path in terms of how often it runs
and how critical is it for the business. The business impact as in, you know, if this is something that has to do with payments or sign-up flows, obviously, we're more sensitive to that.
Kind of like asking ourselves, would we be able to convince the product manager to prioritize it? And we also looked at the risk. If it's a risky change that requires a migration or anything like that, then obviously, it would require more time from the human that's reviewing it. And in that case, we're not necessarily looking for the highest impact ones, but for the highest ROI ones. And because we automate the investigation, we can look at the impact and the risk together and only surface and and require attention on the ones that we believe are the right ones that are worth the engineering time. So, instead of opening ADPRs, we built this human-friendly report that basically tries to convince you that it's worth your while. Something like, "Hey, this endpoint, it's usually taking around 200 ms, but every once in a in a while it takes like 45 seconds, and it happens because you're using distinct and not the search index of Mongo." And there's a very short explanation of what's happening and what's the fix. And then you can either create a ticket and fix it on your own, and create a PR, or just look at it. And we found that building these small gists that are humanly readable and it's easy to understand what's the thing, made a huge, huge difference.
Because we're still living in this hybrid world where humans are in the loop, and we want to respect our place in people's lives and to make sure that we flag things that really matter,
and that we have some confirmation, not only that the fix is good, but also that this issue is worth fixing. And then, when you look at that endpoint and you deploy that change, and all of a sudden it's flat again, it is pretty satisfying. And then next time you'll get that report, maybe you'll have a look, and it will be easier to convince you. So, four things that I learned that could be relevant for you. One is we need to define what matters, and the scoring and the guardrails are what makes this reliable. We can automate a bunch of things, and it becomes easier to just create some slop, but when we start with what are the things that are worth it, even though it's much cheaper to fix these things than it was a few years ago, it's still not free and therefore we need to understand that it's worth the impact the human reviewer and the risk that it entails if it does. Second part is that a lot of what we're talking about today is accelerating developers and what they do and I think what's interesting about this experience is that we automated something that it's not just doing it faster. We're automating a phase that just did not happen in the day-to-day lives of engineers. No one actually stopped every week and had a look on whether there are low-hanging fruits that could be relevant. But now we can use the agents to not only do the things that we're doing faster, but also to help us surface opportunities that we would probably never do without them. This is It sounds simple, but it's hard to actually apply it in the day-to-day, but context over cleverness works almost every time. If they have the right context, if they have the right skills, if they know exactly what these agents get much much more useful and every time a new model comes out, things get better and yet I do believe that at least for most of the cases, the models are good enough already to be able to to automate that. But it is up to us to help steer and
guide to the right directions to make sure that we get the results that we want. They're not necessarily the the absolute right thing because it
doesn't necessarily exist and you know, when we know our domain and our business, we understand which issues matter more. What's worth fixing, what latencies actually impact our customers' experience in the most significant way. And that helps us make sure that we focus on the right things. And in that aspect, agentic engineering is not like coding with an agent. If something works 80% of the time and you're using it with your cursor in your IDE, that's fine because you're there, you're in context, and you can help fix and steer. If we're talking about an automation that runs autonomously, we have to have a very high confidence that we're doing the right thing and that it's not going to just stray off and hand us a bunch of things that we would either waste time on reviewing or just not be able to trust over time. And I think like the hardest part of this automation was to get to a point where we feel confident enough that the issue is worth fixing and that the fix is verified in runtime so that it is handed to a person when we have a fairly good confidence that it actually works. And yet, they still review it, but we know it's worth their time. And getting to that agentic engineering automation level requires crossing towards the 80-90% trust, and it's something that is dramatically different than using an agent directly as an engineer. So, I hope that was helpful. And if there are any questions or anything like that, I'm available and always love to geek out on the AI and the LDC, and uh looking forward to hear about cool automations that you build on your own. Thank you.
