Talk by Benjamin Verbeek of Lovable on how the platform uses continuous self-improvement loops operating at scale (~200,000 projects per day). He describes two loops: one that detects sessions where nontechnical users got stuck and later unblocked, clusters similar cases, and injects that context upstream (with a holdout group measuring project completion rates to prune stale entries), and a "vent" loop where an agent reports platform limitations, missing tools, broken behavior, and confusing docs to a Slack channel. Vent volume spikes served as an incident detector, and a second agent now deduplicates reports and opens PRs automatically. The first hour of the vent tool surfaced a real production bug: filenames with spaces silently failed to copy without appearing in logs. # Video Identity Observations — "How Lovable self-improves every hour — Benjamin Verbeek, Lovable" ## Speaker Self-Identification - **Name and role:** The speaker introduces himself: > "I'm Benjamin van Beek. I'm a member of technical staff of Lovable." - **Background claim:** > "I used to work with satellites, particle physics, and fusion reactors. So, I have a physics background." - **Tenure:** > "I joined Lovable around a year ago." ## Company / Product Self-Description (Explicit Claims) - **Product positioning:** Lovable is presented as a chat-based, no-code software building tool: > "we were actually one of the first... I think we coined the term vibe coding. So, coding without actually looking at code." - **Philosophy:** > "The code was always just an annoying technical layer in between to create what we wanted. But ideally, you just say what you want, you see it, you test it, and you ship it." - **Stated mission:** > "we're building for the 99% who can't code... we are trying to unlock software creation for everyone." - **Scale claim:** > "At that time, we had a few thousand users. Now, we're creating over 200,000 projects per day. That's a significant percentage of all internet websites being created on Lovable." ## Intended Audience - **Conference-talk audience of technical practitioners:** The talk is delivered at a conference event ("thanks to the organizers", "filling up this bonus room"), with an audience polled for product familiarity: > "Do we have any Lovable users in here?" - **Acknowledged mismatch with product's end users:** > "we're building for the 99% who can't code. So, probably not the people in this room, actually." - **Hiring pitch at close:** > "if you think this is exciting, you should join us and help us close this loop and fully automate continual improvements." — indicates the talk doubles as recruitment outreach to AI engineers. ## Claimed Scope of Content - **Two named mechanisms:** 1. **"Lovable Stack Overflow"** — an automated knowledge base built from detecting stuck-then-solved user sessions, with clustering, agent/human review, evals, and context injection with A/B-style effectiveness rating. 2. **A "vent tool"** — the agent sends frustration feedback directly to the team's Slack, leading to automated PRs for detected issues. - **Framing as industry frontier work:** > "the holy grail of AI engineering right now, which is continuous learning at scale." ## Observable Presentation Patterns (Demonstrated, Not Claimed) - **Metrics cited but not shown in detail:** > "the number of messages with fixing intent or people being stuck is dropping significantly" — described as "very very early data." - **Transparency limits acknowledged:** > "There's a few hidden entries here, which I unfortunately cannot talk about." - **Candid tone about failures and learning:** The speaker shares internal stumbles (e.g., > "My first day, GitHub banned us", the copy-tool space/non-breaking-space bug saga), demonstrating a presentation style that includes unresolved-process honesty. - **Employee roles referenced:** > "Our head of product was very excited to see this" — evidence of team-based engineering culture described, not observed directly. - **Recency of automation:** > "It's actually the case now that I just get review requests on my phone from this automation" — claimed personal workflow detail supporting the automation narrative. ## Claims vs. Demonstration Distinction Summary | Type | Examples | |---|---| | **Explicit creator claims** | Coining "vibe coding"; 200k projects/day; mission of unlocking software creation for non-coders; effectiveness of Stack Overflow injection and vent tool | | **Demonstrated by material** | Detailed technical methodology (stuck detection, clustering, eval loops); transparent discussion of bugs, stale knowledge, and review bottlenecks; recruitment-oriented closing | | **Not verifiable from material** | Internal metric charts, hidden model ranking entries, overall product quality impact | # Delivery Observations — Benjamin Verbeek, "How Lovable self-improves every hour" ## Tone & Overall Style - **Conversational, casual register**: Opens informally — "All right. Let's get this party started. Do we have any Lovable users in here?" - **Self-aware humor and irony**: Describes the agent's false fix as "And it was lying." Also frames a risky idea with "This sounds very scary, I have to say, and it sounds like an absolute insane idea." - **Enthusiasm with candor about difficulty**: Pairs excitement with honest admission of problems — "Uh it's also very hard. My first day, GitHub banned us." ## Recurring Phrases & Verbal Habits - **"Very very" as an intensifier** (frequent): "It's very very ancient thing," "very very strong signal," "a very very early data," "very very cheap comparatively." - **Fillers "um" and "uh"** throughout, plus **self-corrections/restarts**: "get back get get past it," "spam spam us," "wanted wanted agent." - **"I think" as a hedge/connector** (repeated): "I think we coined the term vibe coding," "I think that is the world we all want," "And I think in many ways, this is the way we should be building software." - **"In many ways"** repeated: "And I think in many ways, this is the way we should be building software" and "in many ways, this is like building for the future." - **"So" as a narrative pivot** at the start of many explanation turns: "So, you need to learn from that," "So, how do we do that?", "So we have gotten the solution right here." ## Teaching / Structural Habits - **Problem → solution architecture**: Frames a pain point, then presents the mechanism. E.g., "Why do I have to explain the same thing over and over again?" → "in this talk, I want to share two ways how we are working towards this goal." - **Taxonomies and two-way splits**: "I want to split being stuck into two different ways." and "And there there are two classes. There's the things that are just dumb... And then there are the actual hard things." - **Concrete anecdotes as illustrations**: The laggy-website walkthrough ("I'm complaining to the agent about my website being laggy") and the file-copy bug story ("whenever there is a space or raw space in the file name, it's failing to copy"). - **Audience interaction**: Opens with a poll question; later quizzes the room — "do you have any guesses what these spikes are?" - **Emphasis via repetition and meta-comment**: "This loop is incredibly important. I cannot emphasize enough how important this step is." - **Analogies to human behavior**: "You would probably complain to your boss or go venting in Slack. So, we thought, why not do exactly that, but for the agent?" - **Honest disclosure of withheld info**: "There's a few hidden entries here, which I unfortunately cannot talk about." ## Vocabulary Patterns - Colloquial framing of technical content: "vibe coding," "the things that are just dumb," "casting gymnastics," "context rot." - **Milestone metrics used rhetorically**: "we had a few thousand users. Now, we're creating over 200,000 projects per day." - **Workshop vocabulary**: "ship it," "merge this to prod," "we should just ship it," "close this loop." ## Uncertainty Notes - Pauses, filler density, and audience questions appear only in the transcript text; **timing, pacing, and visual delivery cannot be assessed** from this source. - The interaction at the spikes ("Server went down. It's an incident. Yes.") suggests live back-and-forth, but speaker body language or slides cannot be observed. - Some transcript irregularities ("the number of messages with fixing tent," "level Stack Overflow," "Lava") look like transcription artifacts, not speaker word choices. # Position Record: "How Lovable self-improves every hour" — Benjamin Verbeek (Lovable) ## Speaker - **Benjamin Verbeek**, Member of Technical Staff at Lovable. Background in physics (satellites, particle physics, fusion reactors). Joined Lovable ~1 year before the talk. ## Core Thesis - The "holy grail" of AI engineering is **continuous learning at scale**: a mistake should happen once and never again. - Two mechanisms presented: (1) a **"Lovable Stack Overflow"** knowledge injection system, (2) an agent **"vent tool"** for reporting platform issues. ## Claims & Reasoning ### Product positioning - Lovable coined the term **"vibe coding"** (coding without looking at code). - Claim: code is "just an annoying technical layer"; ideally you say what you want, see it, test it, ship it. - Lovable builds **for the 99% who can't code** — "probably not the people in this room." - Users stick with one long-lived project, enabling deeper learning about the user than short-conversation chat agents. - Scale: from "a few thousand users" a year prior to **200,000+ projects per day** — "a significant percentage of all internet websites being created." - Scaling anecdotes: GitHub banned them day one of his employment ("creating too many repos"); they've "taken down a number of cloud providers." ### The "stuck user" framing - Technical users hit friction but work past it; **non-technical users walk away** at the first hard block — "this is what we need to minimize." - Signals of being stuck: repeated identical requests, complaints about implementation, explicit failure reports, abandoning a session. - An **LLM judge** reviews sessions to flag stuck users. - Taxonomy: (a) solvable-with-prompting stuckness, (b) unsolvable stuckness — split into trivially fixable product gaps and genuinely hard problems. - Normative claim: "If it is solvable, then it should work for everyone. And if it is just easy to do, we should just ship it." ### Lovable Stack Overflow mechanism - Detect stuck → not-stuck transitions (not from user giving up) to harvest **high-signal problem/solution samples**. - Ask: what should have been injected at query start to jump to the solution? - **Clustering** prevents overfitting to specific prompts. - An agent reviewer (occasionally human) runs quick evals against example sets. - A lightweight model injects relevant context into the main agent at runtime. - **A/B-style validation**: some injections are sent blank as control; compare project success rates; show solutions more or less accordingly. - Emphasis: this feedback loop is critical because knowledge goes stale with new models and feature changes; must throw away deprecated context to avoid "context rot." - Results claimed: stuck/fixing messages dropping significantly; more deployments (key metric of project completion); all top models in their internal ranking use this knowledge. *(Noted as "very early data.")* - Withheld detail: "a few hidden entries here, which I unfortunately cannot talk about." ### Vent tool (agent frustration reporting) - Analogy: a human without tools complains to their boss or vents in Slack — so give the agent the same outlet. - Acknowledged it "sounds very scary... an absolute insane idea" — "what's even more crazy is that it's kind of working." - **Contrast with external reviewer approach**: external review forces an answer every iteration, yielding low signal-to-noise and overfitting to noise, since most iterations work well; the vent tool fires only on genuine frustration. - Trigger conditions: missing/unsuitable tools, mismatched schemas, conflicting docs, broken platform behavior, repeated environment-caused failures. - Rationale: users often don't know the cause of failure; the agent has direct multi-turn context. - Example vent: agent complaining about Framer Motion TypeScript types (cubic bezier, "casting gymnastics") — Verbeek judged it probably not relevant to the use case but revealing of simplification potential. - **Copy tool bug case**: tool failed on filenames with spaces; ~20 complaints in first hour; initial fix (replace space with underscore) was incomplete — screenshots from WhatsApp/Mac insert **non-breaking spaces** missed by the regex; further special-character reports until properly fixed. "Now this issue never happens again." - Human-relatability benefit: engineers have implicit context to act on complaints quickly. - Evolution: initially worried about spam (kept channel closed); head of product read every message; now a monitoring agent deduplicates, investigates, and opens PRs — devs review and merge to prod; Verbeek receives PR review requests on his phone. - **Incident detection**: spikes in vent volume correspond to platform incidents (sandbox breakages); "it was complaining about right things in general." - Meta-feedback example: agent complained the vent tool made feedback too easy to send and it couldn't retract it — "being ashamed of what it had sent to Slack." - Efficiency argument: inline strong-model intelligence is cheap compared to an external reviewer scanning full context. ## Qualifications & Tensions Preserved - Results data described as "very very early" and since improved — not presented as final figures. - Relevance of agent vents acknowledged as uncertain ("we can ask ourselves is this a relevant vent or not"). - The vent tool idea itself flagged by the speaker as "very scary" and "insane" before being endorsed as working. - Human oversight still required (PRs reviewed by developers "in many cases" merged) — automation is partial, and closing the loop fully is stated as ongoing work. ## Closing Position - Invitation to join Lovable to "fully automate continual improvements" — the loop of detecting shortcomings, merging fixes, and continuous eval is not yet fully closed. # Speaker Attribution & Provenance Record **Video:** How Lovable self-improves every hour — Benjamin Verbeek, Lovable **Video ID:** `01a15ca9-15d9-7d3f-81ce-9af983398e49` ## Speakers & Roles - **Benjamin Verbeek** (self-introduced as "Benjamin van Beek"), **sole identified speaker** — role: **creator/presenter**, described as **"member of technical staff of Lovable"** (talk delivered at a conference with an audience). - Self-described background: satellites, particle physics, fusion reactors (physics background); joined Lovable ~1 year prior. - **Audience members** — brief, unnamed interjections during Q&A-style moments (e.g., answering "server went down" re: vent spikes). No individual identification. - **Mentioned but non-speaking persons:** - **Head of product** at Lovable (anecdote: read every vent message). - **Lovable developers/engineers** (review/merge PRs). ## Referenced Sources & Third-Party Claims - **Stack Overflow** — cited as inspiration for Lovable's internal knowledge system ("very very ancient thing"). - **Framer Motion (Frame Motion)** — third-party library cited in agent vent example (TypeScript cubic-bezier typing complaint). - **Third-party platforms mentioned:** GitHub (banned Lovable on his first day for excess repos/"ripples"), cloud providers (taken down by scaling), WhatsApp and macOS (screenshot non-breaking space behavior), Slack (vent feedback channel). - **Internal Lovable systems described by the speaker** (unverifiable claims, no external source): LLM-judge stuck detection, "Lovable Stack Overflow" knowledge bank with clustering, external agent/human reviewers, injection A/B testing with blank-injection controls, "vent" feedback tool, automated PR pipeline. - **Metrics claimed by speaker (unverified):** ~few thousand users at hire → **200,000+ projects/day**; early data showing drops in "stuck" messages and increased deploys; ~20 vent complaints in first hour of the vent tool. - Speaker notes some **hidden entries in internal model rankings**: "There's a few hidden entries here, which I unfortunately cannot talk about." ## Promotional Segments - **Closing recruitment pitch (flagged):** "if you think this is exciting, uh you should join us and help us close this loop" — hiring promotion. - **Product promotion throughout:** Extended positive framing of the Lovable platform (vibe coding, claiming the company "coined the term"), scale claims, and success claims for its self-improvement loops — promotional in character though presented as a tech talk. ## Availability & Limits - No curated metadata supplied beyond title. - **Comment availability: not stated.** - Quantitative results shown as "very very early data"; presenter withholds some ranking details (hidden entries). - Single-speaker talk; all claims are first-party (Lovable) with no independent corrobination in source. All right. Let's get this party started. Do we have any Lovable users in here? Good, good, a few. Very nice. A warm welcome to to this talk on how Lovable is improving every hour, including this hour as I'm speaking. I'm Benjamin van Beek. I'm a member of technical staff of Lovable. It's so amazing that we are filling up this bonus room. I think we're around three times as many people as would have fit in the original. So, thanks to the organizers. Um This is my background. I used to work with satellites, particle physics, and fusion reactors. So, I have a physics background. And today, I'm working at Lovable and working towards what is maybe the holy grail of AI engineering right now, which is continuous learning at scale. How do we learn from mistakes? So, you've probably all experienced working with an agent and having the feeling, "Why do I have to explain the same thing over and over again?" This is what we want to avoid. We want to have a mistake happen once and then never again. So, you need to learn from that. And in this talk, I want to share two ways how we are working towards this goal at Lovable. This is the Lovable interface. Uh we were actually one of the first uh I think we coined the term vibe coding. So, coding without actually looking at code. So, you have chat interface where you describe what you want to happen, and you have a model sandbox, for example, where you can see directly what you're creating. And I think in many ways, this is the way we should be building software. The code was always just an annoying technical layer in between to create what we wanted. But ideally, you just say what you want, you see it, you test it, and you ship it. We have some nice benefits with the Lovable platform, which is that people stick to one project for quite a long time. This chat can go on forever, essentially. And people have like this one artifact that they are really passionate about and they want to ship. So, we can learn a lot about this one thing versus, for example, a chat agent where you have very, very short conversations. We can actually get to know the user in quite detail. Another exciting thing is that we're building for the 99% who can't code. So, probably not the people in this room, actually. There are still quite a lot of people, evidently, who use our product, but we are trying to unlock software creation for everyone. In the future, you will not have to look at code to be able to create software. I think that is the world we all want to move towards, and we are trying to do that right now. And in many ways, this is like building for the future, I think. So, a reason that startups succeed with new technology is that they are a bit naive, and they don't really think about all the issues that can happen. They don't think too much about all the details in implementation. They just go until it works. This is how I like to see our users. They just go until it works, and they're not sort of held back by old paradigms. This is a wonderful group of people to to build for. Um and it scaled quite rapidly. Uh I joined Lovable around a year ago. At that time, we had a few thousand users. Now, we're creating over 200,000 projects per day. That's a significant percentage of all internet websites being created on Lovable. And that scaling journey has been incredibly fun. We can do so many cool things now, like the things I will talk about in this talk. Uh it's also very hard. My first day, GitHub banned us because we were creating too many ripples. And we've taken down a number of cloud providers along the way. How do we succeed with AI? Um, I like to think about it in this way. Technical personas generally have this amazing moment with AI where they are just building and building. You're accelerating 10 times, 100 times faster than you used to. But sometimes we hit some friction points, this yellow part, and you have to intervene or like from the bit harder. And sometimes you even get stuck. Maybe you have to manually go in and change a config somewhere. You need to really do a lot of manual work, change the setting, add an end bar, API key, etc. And then you move on and you really get the good parts of AI and you can work past the bad parts. You're still annoyed by it, but you can work past it. And non-technical persona is not really like this. They might prompt their way past the first friction, but as soon as this technical block happens, they generally walk away and they give up and they actually still never experience successful AI. This is what we need to minimize. We can never get stuck to a point where a non-technical user cannot get back get get past it. And this is a very exciting problem to work on. Luckily, models are getting better and the requirements of these people is generally to a point where we can achieve them. But for the past year, we have been working on making sure none of these red bars happen. People can never get stuck. How do we do that? First of all, we want to define like what does it mean to be stuck? We probably want to find those cases and then learn from them. So, one clear signal of someone being stuck is that they ask for the same thing more than once. Probably stuck or it didn't work as smoothly as they could. Maybe they're complaining about how something was implemented or that it failed very explicitly or they gave up on a session that they would otherwise have continued. And we can have a LLM judge that is just looking at sessions and trying to look for these situations and they can flag it and say, "Hey, I think this user is stuck." I want to split being stuck into two different ways. You can be stuck in a way where it's possible to solve. This is maybe the yellow part that I showed. And you can get past it with the right prompting. Some users are more willing to go put in that effort and get past that point. Others will give up, but it is possible to solve with the current structure of our product. And then there are tasks that are not solvable even if you say all the right things. Maybe we're just not supporting it for some reason. Um And there there are two classes. There's the things that are just dumb that we don't support. Maybe it's a bug. Maybe it's a very very simple thing we could add to the product. Uh And then there are the actual hard things that would take like weeks of engineering AI-assisted engineering effort. These first two things we should be able to improve on very rapidly, I think. There's no excuse to not improve on these things. If it is solvable, then it should work for everyone. And if it is just easy to do, we should just ship it. So, how do we do that? Um some of you might have heard of Stack Overflow. It's very very ancient thing. We had an idea to build a Stack Overflow for Lovable. So, we basically learn from uh whenever someone is stuck and has an issue, we try to figure out the solution and give that to the agent. This could look something like this. Um I'm complaining to the agent about my website being laggy when I scroll. It's bad performance. Maybe I'm not stuck yet. This is just me complaining about something and giving it a new task. The agent probably replies something like, "I fixed it. Maybe I quantized the animations to make it more performant." And it was lying. The agent has failed and it actually made things worse. Now the website is jumpy and laggy. Terrible. Okay, this is a clear case of someone being stuck. They asked the same thing again. They probably wanted wanted agent to try again. And they're complaining about an implementation that didn't do what they wanted. This might repeat for a while where they keep iterating several times. Some people give up at this stage, but at least some people will continue. And eventually the agent might find the true solution. Which in this case happened to be that overlay text had individual gradients which made the animation super low performing. And now we see that is stuck changes back to false. It succeeded. So this we can flag. We can notice whenever it went from being stuck to not being stuck. And ideally not because the the user gave up. This we can flag and now we have a high signal sample of a problem that was high friction or not solved that was solved. So we have gotten the solution right here. And the question we just ask is what should we have injected at the start of this query to jump straight into the solution so the next user does not experience this friction. What we do is we create a new lovable Stack Overflow knowledge entry. And in general we actually do some clustering at this step where we look for similar issues. We look for what is the actual information we should give so that's not overfitting to the specific issue. We don't want like a million overflow pages Stack Overflow pages that are all talking about if you get this exact prompt then you should do this exact thing. It's not very helpful. We do some clustering. And then we have an external reviewer, generally an agent and then maybe in some cases we have a human if we are uncertain, but in most cases it's actually just an agent that generates and runs a quick eval on this and sees if this did this resolve the specific examples that we had in the set. From that we get a full bank of level Stack Overflow problems and solutions that's being continually updated. Whenever the model is working, we have a lightweight model that tries to inject that context when needed. If it detects there is an issue and there is an answer, it injects that context into the main agent. And sometimes it detects that I should inject this, but we inject a blank. So, we don't actually send anything for a small sample of use cases. And this allows us to with very high signal review whether this solution was actually useful in production. And we rate it. We compare the the group of projects where it was injected and where we it could have been injected, but it wasn't. And we say which of these projects were actually more successful overall. It was more successful, then we show it more. And if it was less successful, we should show it less. And this loop is incredibly important. I cannot emphasize enough how important this step is. Because things are moving around this set of knowledge all the time. It gets stale whenever a new model is released. It gets stale whenever we change features. It gets stale incredibly quickly. So, very often, we have to rebalance this, but we also have to throw away a lot of this context. And this allows us to really be at the frontier of what is solvable right now, but not have a lot of old deprecated knowledge that is giving context rot, um and also in many cases just hampering the actual uh agent. And this is working at scale. Uh this is a very very early data actually, where we're doing quite a lot better now. Um but the number of messages with fixing tent or people being stuck is dropping significantly. And we have also significant number of people that deploy more. This is one of our key metrics. That they actually finish a project. This is a very very strong signal that it has worked all the way through. They've never gotten stuck so bad that they give it given up and abandoned the project completely. Um we also do internal ranking with this. So, it's really interesting to see how the models perform on this set of problems that we have collected. And with the Stack Overflow information, uh all models in the top of our ranking use this information. There's a few hidden entries here, which I unfortunately cannot talk about. Um but it really really makes a significant boost in our internal ratings. And then there's a second set of uh being stuck, which is uh when it's not actually solvable with the current setup. Maybe there's a bug in our product. Um but in these cases we think it should be easy in principle. And what happens if you think something should be possible, but it just doesn't work? You feel significant frustration. And what would humans do if this happened? Well, if you were given a task and you just don't have the tools to do it, you would probably complain to your boss or go venting in Slack. So, we thought, why not do exactly that, but for the agent? So, we basically are asking Lovable, "How are you doing? Can we help you in any way?" And we gave it an outlet to let out its frustrations. So, we asked ourselves, "What if we could uh let the Lovable agent give direct feedback to its creators?" This sounds very scary, I have to say, and it sounds like an absolute insane idea. Um what's even more crazy is that it's kind of working. This kind of detecting issues is something I've heard during this conference, but also a few other times, where you might have an external reviewer looking at the conversation and asking, "What could have been done better to reduce friction?" One problem with that is that you get a pretty low signal-to-noise ratio. Which you're basically forcing an answer on every iteration. And the reality is that most iterations actually just work pretty well. And you will overfit to noise. In this case, it's prompted to only send feedback if it is really frustrated. And you can tune that balance until you get a lot of signal. So, we gave our agent a vent tool. A way to complain to its creators. Um it looks something like this. It's a vent send feedback tool. You should use this when tooling docs or platform behavior material is low so it degrades your work. For example, missing or unsuitable tools, unclear tool names, parameters or schemas that are not matching what you were expecting, confusing or conflicting docs or instructions, broken or unexpected platform behavior, repeated failed attempts at causing caused by environment limitations. So, basically a list of all the things we think it should we should be able to solve and that we hope is not an issue, but if it is, please tell us. And our users generally don't know what the cause of a problem is. When I'm using Lava when I get stuck, I generally don't know. But the agent actually has a lot more context. It has literally been working on the issue, sometimes several turns. And it generally has a lot of context on this problem. So, it can look something like this. It actually gets sent directly to our Slack. And the agent says something like, "I'm so annoyed at Frame Motion's TypeScript types for generating I think this is a cubic bezier. I just want to be able to send a list of four numbers. That should be fine. That's all I'm sending. I don't need all this casting gymnastics." Now, we can ask ourselves is this a relevant vent or not? Um but in this case, it it probably just was not relevant for its use case and we could have simplified things a lot. Another benefit of this setup is that it's very easy to understand as a human. It's very relatable that you're complaining about your workflow and that means that engineers sort of already have all this implicit context on how to alleviate the issue. Another problem that got reported by the agent was our copy tool was struggling with certain file names. It couldn't copy it to another place where we store store documents. And we were so confused by this. Like we checked the tool, it's working. Um we didn't even know that this tool was failing so regularly until the agent told us. And it said whenever there is a space or raw space in the file name, it's failing to copy. And we got like 20 complaints about this in the first hour of launching this tool, which is crazy. It turns out that there was an error where it could not copy files with a space in their name. We fixed it and we told it whenever there's a space, just replace it with an underscore. And we kept getting the reports. And then we realized that when you screenshot something in WhatsApp or on Mac, it it enters a non-breaking space. Which we did not replace in our regex. And this kept complaining for various other special characters until we did solve it properly. And now this issue never happens again. I think this is an a prime example of something that's pretty hard to detect in other ways. You can see the tool failure, but this is just a such a clear case of how to fix it. And we just do it. I will skip this one. Um It's essentially this is the flow. The agent experiences an issue, it uses the event tool, it's sent straight to Slack. At first I was very uncertain if this would work. And I didn't want to spam spam us. It was like it closed down channel. I didn't invite so many people. Our head of product was very excited to see this. Uh he was reading every single message. Now it's a bit more balanced and we actually have an agent that's monitoring removing duplicates and investigating and creating a PR for all these issues all the time. And we're still at the point where devs are reviewing and then in many cases actually merging this to prod. This is number of events over time. Uh do you have any guesses what these spikes are? Why do we see spikes in the number of events cost per time? So, that's server went down. Server went down. It's an incident. Yes. Something broke in the platform. At some point, our sandboxes broke. Something something else broke. And the agent is very upset about this. It's complaining a lot. So, it turned out that this was actually a prime place to notice when our product was having an incident. And it actually gave a pretty good sense for what the problem was. It was complaining about right things in general. Um very brief I'll keep this brief. Um you get the strong model intelligence versus an external reviewer. You generally don't want top frontier level intelligence to look through a lot of context. But if it's in line, it's very very cheap comparatively. Um this was an example where the agent was actually giving meta feedback on the venting tool. It said it's too easy to send feedback and I can't pull it back. It was being ashamed of what it had sent to Slack. Um and it's actually the case now that I just get uh review requests on my phone from this automation to hey, review this PR that was completely automated. I look through it quickly and we can merge it. And we're continuing to work to close this loop of detecting a shortcoming, merging a fix, and then continuously review and eval that. And if you think this is exciting, uh you should join us and help us close this loop and fully automate uh continual improvements. Thanks for listening.