Talk by Annabell Schäfer (Langfuse) on why agentic self-improvement loops require domain expertise and well-defined target functions. Using a case study of an auto-improvement loop on arXiv paper classification evaluated against a ground-truth dataset, the talk covers: setting up a minimal self-optimization agent, iteration steps and stopping criteria, results showing a 15% accuracy improvement, why coding's clear "compile" signal set a false precedent for other domains, and how human-agent collaboration and data review fit into continuous improvement. # Channel Presentation Observations: Annabell Schäfer, Langfuse ## Explicit Self-Description - **Speaker role**: "I'm a growth engineer at Lenfuse [Langfuse]" - **Company/product claim**: "we're the largest open source observability and evaluation platform for your AI system" — a direct creator claim about the company's market position - **Call to action**: "if you want to know more about Lenfuse and also how we think about these topics, you can find more on length.com [langfuse.com]" — confirms the video serves partly as a product/company touchpoint ## Claimed Scope - **Topics covered**: AI application evaluation, self-improvement loops, prompt optimization, evaluator design, and domain expertise encoding - **Evidence**: The talk spans "how you should stop burning your tokens," building "domain expertise early into your loop design," and a detailed case study on "minimal self-optimization loop" with a classification task - **Practical scope**: Advice-oriented content aimed at teams building AI applications, e.g., "what we see working is really looking into what are the quality criteria that you want to work for your application" ## Intended Audience - **Developers and AI application builders**: References to "Boris Journey," "Peter Steinberger," and "the whole Kapathy auto research and auto improvement topic" assume familiarity with AI engineering discourse - **Teams deploying AI in regulated/vertical domains**: Cites "medicinal compliance or healthcare or all kinds of chatbots," "your vertical domain expert AI agent," and "automation of requirements extraction in medicine device compliance" - **Practitioners responsible for production quality**: Advice like "you might hit production, review this data... review it as a human" targets engineers shipping and maintaining systems ## Demonstrated Presentation Patterns - **Conference-talk format**: Opening "Hi everyone," closing "Thank you so much for listening," and "super excited to meet some of you at some point in person" indicate a live presentation - **Product-integrated case study**: The experiment is demonstrated "within length view [Langfuse]" with "a screenshot from our platform where we managed the prompts" — the product is shown as the tool of the work, not merely advertised - **Data-driven narrative**: Specific metrics cited throughout — "200 items in a fit data set, 100 in a validate data set, and 300 in a test data set," "15% uptake from our baseline," "68% accuracy" rising to "83%," "80.2%" on test - **Named-tool specificity**: References concrete models and tools — "GPD5 for nano [GPT-5 nano]," "cloud code [Claude Code]," "cloud us 4.8 [Claude Opus 4.8]" — demonstrating a technically specific style - **Evidence-from-customers framing**: "we at length [Langfuse] see that the teams who are investing heavily here... are the ones who manage to continuously upgrade and improve their application" — positions the channel as drawing on observed customer patterns (a claim, not verifiable from the talk itself) ## Boundary Notes - The "largest open source observability and evaluation platform" claim is the creator's own and unverified within the material - The transcript is auto-generated with transcription errors ("Lenfuse," "GPD5," "cloud us 4.8"); no private motivations beyond stated professional promotion are inferred # Delivery Observations: Annabell Schäfer — "Stop Burning Tokens" ## Tone & Overall Register - **Conversational, engineer-to-engineer tone**: frequent filler words and hedges throughout ("um," "uh," "I mean," "yeah") — e.g., *"Um yeah, I mean overall we can see here that there's a a tendency to increase."* - **Self-deprecating honesty about her own expectations**: *"I was surprised I would have probably um spontaneously added descriptions to the labels. Uh but the prompting guide or the model together with prompting guide um decided that actually the right approach is here and I mean it worked. So uh I'm not going to going to interfere here."* - **Warm, appreciative closing**: *"Thank you so much for uh listening to me. Um very excited that you took the time... I'm super excited to meet some of you at some point in person."* ## Teaching Habits & Structure - **Concrete-example-first teaching**: after framing the abstract problem ("target functions are not nearly as clear"), she grounds it in a worked example — a paper-classification task with defined fit/validate/test splits. - **Contrast framing**: repeatedly sets up a clear vs. unclear case — *"does the code compile"* vs. *"barely any deterministic yes no target functions in most cases."* - **Explicit step-by-step walkthrough of the loop**: "first you run the base prompt... then the uh then float proposes an update... and then reruns and only um accepts if this uh also improves on the validation set." - **Advice list at the end**: three recommendations — work with experts, review data as a human, "think about the whole thing as a system that generalizes." ## Recurring Phrases & Vocabulary - **"Burning tokens"** as the central motif, from the title through the closing: *"make sure that you're not um just burning tokens but actually looping towards um a system where a human and your agent together collaborate."* - **"High signal feedback"** as a repeated key term: *"creating these high signal um feedback loops"*, *"having the volume there and having the high signal feedback um oops are the foundation."* - **"What does good mean"** — appears twice: *"what are the quality criteria that you want to work uh for your application. So what does good mean?"* and later *"define what good means."* - **Repeated "yes / no" framing** for evaluators: *"a very clearcut yes or no,"* *"is the true label equal to the predicted label,"* *"categorization yes no."* ## Communication Patterns - **Cites named figures and trends as context**: *"Boris Journey saying he doesn't write any prompts anymore... Peter Steinberger being like you should be designing loops... the whole Kapathy auto research and auto improvement topic."* - **Numeric specificity**: precise dataset sizes (200 fit / 100 validate / 300 test), accuracy figures (68%, 83%, 80.2%, "15% uptake"), and stopping criteria ("15 runs uh completed or we have 92% accuracy"). - **Caveats attached to claims**: *"depending on the complexity of your application and the steps in there of course it might differ."* - **Acknowledges advice that is common but still endorses it**: *"this is like repeated advice and everyone's saying it and everyone's saying talk to your customers. Um but here it's something you can actually start to encode."* - **Relatable micro-anecdotes from other companies**: *"I've also seen companies that have English names that are checking for did we not accidentally translate our name to Spanish?"* ## Uncertainty & One-off Moments - **Speculation flagged as speculation**: *"which probably in most real cases um yeah it takes some time to get there"* — hedged rather than asserted. - **Isolated moment**: an audible *[snorts]* mid-presentation ([~730]); a single occurrence with no observable pattern. - **Disfluency clusters around topic transitions** ("So um," "Okay, so") appear consistently enough to look stable, but density varies and could be transcript-artifact dependent. # Position Record: "Stop Burning Tokens: Why self-improvement needs domain expertise first" **Speaker:** Annabell Schäfer, growth engineer at Langfuse (open-source observability/evaluation platform for AI systems) ## Core Thesis - Teams should **build domain expertise into loop design and application design early** rather than "burning tokens" on auto-improvement against weak signals. - Loops (citing Boris Journey's "I don't write prompts anymore, I have loops," Peter Steinberger's "design loops, not prompts," and Karpathy's auto-research/auto-improvement discourse) come mostly from a **developer perspective**, where coding has a clear target function: **does the code compile**. - In other domains (medical compliance, healthcare, chatbots), target functions are **not nearly as clear**, and any target given to an agent is **inherently incomplete** — "you might initially think you're trying to head down here but actually your optimal destination is up there." - Teams investing in defining target functions and building good evaluators are the ones (per Schäfer's observation at Langfuse) that continuously improve and **ship with confidence**. ## Experimental Setup (Schäfer's experiment) - **Goal:** Find the clearest-cut target function to study the role of target functions in auto-optimization. - **Chosen task:** Single-label classification of archived papers by title/abstract, with author-assigned ground-truth labels. - **Data:** 200 fit, 100 validate, 300 test items — split chosen to guard against **overfitting**; test set deliberately not linked to the agent. - **Agent:** Simple flat-list prompt on **GPT-5 nano ("for nano")** — chosen to test whether a very small, cheap model can self-improve, since good models "got really, really good but also very expensive." - **Optimizer:** Claude Code with **Claude Opus 4.8** proposing prompt updates, referencing the GPT-5 prompting guide and a task.md describing the loop. - **Base prompt:** Deliberately minimal (flat label list + "classify this paper with a label") — no prose guidance — to see how the system handles the weakest starting point. - **Loop:** Baseline run → per-item scoring → error-cluster analysis on fit set → hypothesis-driven prompt update for the biggest error category → rerun, accept only if validation improves. - **Stopping criteria:** 15 runs or 92% accuracy. ## Results (as reported by Schäfer) - **Baseline 68% accuracy**; jump to **~83% by the third/fourth iteration**; plateaued around **80%**; overall ~**15% uplift**. - Generalized to **80.2% on the unseen 300-item test set**. - **Biggest gain (~10%) came from the very first iteration**; subsequent movement was small. - Qualification she preserved: the "clean" labels turned out **not fully clean** — authors had "creative freedom" in choosing labels, and their in-the-moment reasoning wasn't recorded. ### What the optimizer did - Added: a general **classification approach**, rules for **disambiguating similar classes**, confusion-pair patterns, and **examples** of frequently misclassified pairs; preference for the more specific over the broader label. - Schäfer's stated surprise: she "would have probably spontaneously added descriptions to the labels," but the model + prompting guide chose rules + examples instead — "it worked. I'm not going to interfere here." ### Why the first jump worked (her analysis) - Quantifiable, reliable failure mode from **yes/no right-wrong** signal; 64 identified errors; dominating patterns and biggest confusion identified; clear hypothesis formed. - Sufficient **data volume** (200 items, only 10 labels, so each label covered enough to yield signal). ## Translating to Real Domains - Generic evaluators like **correctness, helpfulness, hallucination** are popular but **low signal** for auto-improvement — often scored 0–1 or 1–5 with undefined criteria, making them **inconsistent across runs** (LLM-as-judge is nondeterministic). - Instead, define **domain-specific, high-signal, yes/no quality criteria**, e.g.: - "The answer is based on the knowledge base: yes/no" (check internal-info snippets against retrieved context) - Brand-voice checks (name spelled correctly; not accidentally translated into Spanish) - **Categorization of known failure modes** ("which of these five failure types happened?") ### Volume requirement - Thousands of examples were available; 200/100/300 already yielded strong feedback; needed volume depends on application complexity — **volume + high signal feedback are the foundation**. ## Recommendations 1. **Work with domain experts** — encode expertise into concrete examples, run sample runs together, ask "why is it like this here?", surface implicit knowledge, identify failure modes, define "good," and co-create high-signal evaluators. 2. **Review production data as a human** — not only with coding agents; scope and failure modes shift over time; catch failures before customers do. 3. **Build the system to generalize** — include real **validation mechanisms** (as in traditional ML), clear loop instructions, and an **escape hatch / stopping criteria** so the system doesn't run for hours "hitting a wall and burning tokens." - End goal: a system where **human and agent collaborate on improving** the application. ## Tensions / Qualifications Preserved - Coding's "compiles" signal is clear but **insufficient** ("compiling doesn't mean it's great") — she does not claim it's a perfect target function. - Her chosen "clearest-cut" target function turned out **noisy** due to author label subjectivity. - General evaluators are framed as having been "a good thing to go for" early on — she critiques them specifically **for auto-improvement**, not universally. - Her own prompt-engineering instinct (label descriptions) was overruled by the optimizer's approach; she reports deferring rather than intervening. # Video Attribution Record ## Video - **Title:** Stop Burning Tokens: Why self-improvement needs domain expertise first - Annabell Schäfer, Langfuse - **Video ID:** `01a15ca8-79c2-7efb-a6d7-74ce29c15935` ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Annabell Schäfer** (introduced as "Annabelle") | **Creator / Presenter** — self-identified as "growth engineer at Lenfuse [Langfuse]" | Opens: "Hi everyone, my name is Annabelle. I'm a growth engineer at Lenfuse..." | - Single-speaker presentation (talk format); no guests, narrators, or other speakers appear in the transcript. ## Referenced Third Parties / Sources (as cited by the speaker) - **Boris Journey** — quoted claim: "he doesn't write any prompts anymore. He has loops" (paraphrase by speaker, June 2026 context). - **Peter Steinberger** — quoted claim: "you should be designing loops and not prompt your agents" (paraphrase by speaker). - **Karpathy** ("Kapathy") — referenced re: auto research and auto improvement topic. - **GPT-5 for nano** ("GPD5 for nano") — model used in the experiment's agent/classifier. - **Claude Opus 4.8** ("cloud us 4.8 eight" / "clide oppus 4.8") — optimizer via Claude Code, described as "one of the frontier models." - **GPT-5.4 prompting guide** ("GPT54 prompting guide") — context reference used by the optimizer. - **Classified archive papers dataset** — papers with author-assigned primary labels from title/abstract (200 fit / 100 validate / 300 test items). All third-party claims are attributed to the speaker's own recounting; no direct external quotes appear. ## Provenance Facts - Speaker's affiliation: **Langfuse**, described as "the largest open source observability and evaluation platform for your AI system" (speaker's own claim). - Experiment described: minimal self-optimization loop with single-label classification; results reported (baseline ~68% accuracy, first-iteration jump to ~78%, plateau ~80%, test-set generalization 80.2%) — all figures come from the speaker's account, not independently verified. ## Promotional Segments - **Opening** — speaker introduces herself and describes Langfuse's offering ("largest open source observability and evaluation platform"). - **Closing** — explicit promotional call-to-action: "if you want to know more about Lenfuse and also how we think about these topics, you can find more on length.com" (length.com / langfuse.com as spoken). ## Availability & Uncertainty - **Comment availability:** No comment data supplied; comments status unknown. - **Curated metadata:** None supplied beyond title and transcript. - **Transcript quality:** Contains apparent speech-to-text errors (e.g., "Lenfuse," "cloud us 4.8," "Kapathy," "length.com"), noted without correction beyond context; speaker's name given in title as "Annabell Schäfer," self-introduced only as "Annabelle." - **No timestamps** available in transcript. Hi everyone, my name is Annabelle. I'm a growth engineer at Lenfuse and we're the largest open source observability and evaluation platform for your AI system. And today I'm going to share about how you should stop burning your tokens and why you should start building in domain expertise early in into your loop design, but also into your overall um application design to make sure you're continuously improving and updating your application. It is June 2026 and the whole internet is about uh loops right now. So we have Boris Journey saying he doesn't write any prompts anymore. He has loops and Peter Steinberger being like you should be designing loops and not prompt your agents. And I mean the whole Kapathy auto research and auto improvement topic blew up earlier this year already. But all of them are coming a little bit from this de developer perspective. And um coding is one for for a good reason. One of the cases where this whole um automatically reaching a goal uh worked quite well because they've always had a at least one um target function that wasite quite clear. And this was uh does the code compile or not. And of course just because code compiles doesn't mean it's great and doesn't mean um that all the features are exactly how you wanted them. But you at least know that you shipped something that worked. And of course this was then expanded over time to to expand this target function. But essentially it's a does it work or does it not in all other fields and especially if you're building AI applications for some kind of domains like medicinal compliance or healthcare or all kinds of chatbots these target functions are uh not nearly as clear and a target that you give an agent is actually also always incomplete. So um you might initially think that you're trying to head down here uh but actually your optimal destination is up there and to get there and to understand this it takes quite some time and to figure this out and it's yeah it's just inherently a difficult problem especially if you are in a field where a clear yes or no like does the code compile um doesn't really uh cut it. So it is very difficult but at the same time we at length you see that the teams who are investing heavily here in the middle so uh making sure they capture what they actually want to work so the target function and build this out and make sure they have good evaluators that are evaluating this are the ones who manage to continuously upgrade and improve their application over time and also ship with confidence because if you know um it's working as you intended to h then you can also sleep um well at night in case you push a code change. Okay, so now we know it's super important and you should be doing it, but at the same kind it's time it's almost impossible to actually h get it right. Uh so we were setting out and wondering what's the clearest cut target function we can find um for an agent to use and also if you run an auto optimization on it h what can we learn about the role of target functions from from running it on it and the clearest cut target function we could find was a single label classification task that has has a very clearcut yes or no. So for example, let's say we have an item to categorize. There's a true label. For example, oh, it's an order. And then we have a set of available labels, order, complaint, inquiry. And our classifier then assigns um one label. And you can very clearly say is the true label equal to the predicted label. And if yes, um then this one is right and the next one might be wrong. And overall you can calculate an accuracy value and get a very clear signal from here. So we put this target function together with an agent and also with an optimizer. And how this minimal loop looked like I'm going to share with you now. So here's our minimal self-optimization loop. Up there in the first two rows we can see our target function. In our case we opted for classified um archive papers. So it's a set of papers that based on their primary um based on their title and abstract h got a primary label from the author um to categorize it for the other people in research. And um yeah, we're going to uh we have the the ground truth here and we have it in 200 um items in a fit data set, 100 in a validate data set, and 300 in a test data set. Um just to also make sure we're not overfitting. Then we have our agent, which is more or less just a a simple prompt. um uh based on GPD5 for nano because we wanted to see how a very small and cheap model performs on the auto improvement because the good ones actually got really really good but also very expensive um over time and we have this flash list of labels and then we have our optimization process that runs through uh cloud code leveraging cloud us 4.8 eight. So one of the frontier models and it proposes the prompt updates and has this context reference the GPT54 prompting guide as well as a task MD that is describing the loop. Looking a little bit closer on how this looks like. Here we can see our target function in this case especially our fit data set uh within length view. So we see an input column where the title and the abstracts are inside as well as an expected output column. So what kind of uh label should be applied and um on the those three different data sets the the idea is on the fit data set you run it you look for the errors and especially error clusters and um opus should then formulate hypothesis for prompt updates in the validation set we then check if those prompt updates actually also generalize to unseen data and then finally when we're like reaching a plateau or our stopping criteria um then we're running it on a test set to see how well we actually um generalize for untouched data that was not part of the training process. Our base prompt is a very flat list of labels and just a simple task classify this paper with a label. Um of course if uh if we would write the prompt we would probably add some pros how to think about it and all of this. But we just wanted to see what happens if we use the very base version um of this prompt and how the the system is dealing with it. And our leap loop is a step-by-step instructed uh loop um done through a task markdown file. Uh so the overall idea of the loop is first you run the base prompt on the fit and in the validate set to get like a baseline accuracy. You score it per item and overall then on the fit data set you do this error analysis. So you really look into what kind of categories frequently are done right? what kind of um item pairs are maybe often confused and also what's then the uh underlying abstract and title um belonging to it. So we can maybe find patterns there. Then the uh then float um proposes an update uh for the biggest error category and publishes a new prompt change and then reruns and only um accepts if this uh also improves on the validation set. And we have two stopping criteria. Either it's 15 runs uh completed or we have 92% accuracy reached. And as soon as this is happening or um it stops for some other reason, then we perform a final uh run on a test data set to see it. So yeah, here on the right you can see all of this is described in a markdown. We point at the data sets and we not even give it the test data set link. Um so it will not look into it before we wanted to. So what happened? Um yeah, I mean overall we can see here that there's a a tendency to increase uh but also we can see that um yeah we reached a 15% uptake uh from our baseline which is actually quite solid. Um the first run ended at uh 68% accuracy and on our third fourth iteration um we went all the way to 83% which then also afterwards um plateaued a bit. Overall it kept this level around 80% and we wondering okay actually we wanted the perfect clearcut target function. Uh but then looking deeper into the data and how those labels are chosen we realize okay there's um some creative freedom for the authors to choose what kind of label they want. Uh so even though description might make sense uh they might have chosen in that moment that they actually want to go for a different label and that's then also of course not recorded in here. And we also saw this uh improvement then generalized to the um test data set on yeah to 80.2%. Um so the generalization was also on the 300 item unseen data um quite good and uh in a way that you could say okay this actually worked and um the most interesting thing uh we thought is that the first iteration immediately gained 10% and then it was only a little bit movement. So uh it somehow got a lot of information from this very first run already and made a made a big uptick uh that we can can see here um up until the yeah very close to the final result also and we um wanted to look closer into what it actually did here. So here we can see on the left side the um flat label list prompt. So here on the left again a screenshot from our platform where we managed the prompts and on the right side we can see what happened. So the very first thing it um added is a general classification approach. So how should the model think about um the behavior of uh choosing one of these labels? So when what's what's the main goal of doing so? Then there's some information on how to decide between two very similar classes and then also some um related confused patterns. So it's uh basically telling it um when to choose which and under which conditions and how to go about it and to rather go for the more specific label than for the more broad label because there's some implicit testing in the label structure as well. And then it also added some um examples for the the item pairs that it um yeah frequently missed or frequently mclassified. So this is what uh we can see here. Uh so overall the approach was um improvement loop added rules and examples and I would have probably I was surprised I would have probably um spontaneously added descriptions to the labels. Uh but the prompting guide or the model together with prompting guide um decided that actually the right approach is here and I mean it worked. So uh I'm not going to going to interfere here. Um yeah what we wanted to dive deeper a bit more then is what actually happened in this one 10% uptick uh that we have. So let me remove myself a little bit because we need the bottom right corner. So this is the um the reasoning step that clude oppus 4.8 took uh in this one jump from 68% to 78%. And what we can see here is that it has a very clear okay we have 64 errors. These are the dominating patterns. These are the number one answer sync and this is the biggest confusion. So also what are the two labels that are most frequently confused and where there's like um issues coming up. And then it's forming a hypothesis and saying okay this loop will extract ex exactly address this pattern. So it got a very clearcut um clearly quantifiable and reliable failure mode because they were just wrong on a yes no right wrong basis. Um they also had it had a lot of data to look at. we had a 200 item um fit data set which probably in most real cases um yeah it takes some time to get there and to also like make sure that it was only 10 labels so each of them got covered like um enough times that that it can get a signal from it um yeah and like this it could form a datab hypothesis and ended up uh returning a 10% gain on the first uh run and then I mean it was a bit of movement and we also found a better version then uh but the biggest jump was uh just the very first one from a very clear-cut signal, we could have probably stopped there and already have very good baseline. So um given this right wrong high signal feedback works really really well on this classification test but also knowing that this is not um the case for all others like there's barely any deterministic yes no target functions in most cases you will run it this time and next time you get a different um next time you run the same evaluator you get a different answer from the same kind of evaluation you ran um just because yeah LLM as a judge is also the nondeterministic we were wondering how can we translate this right wrong high signal feedback into other applications. So how can we try to shape this in a way it also works for your um vertical domain expert AI agent or for your uh automation of requirements extraction in medicine device um compliance or whatever your use case might be. \[snorts\] And um for this um we overall see the approach working that um while the market for for a good reason likes those um evaluators like correctness or helpfulness or hallucination which like early in the days um were were a good um thing to go for. Um this is actually if you're trying to um autoimprove against this a rather low signal especially also because it's put on a often on a scale out of like between zero and one and 10 one and five and for this to really work properly you need to define each of those numbers. What does it mean in which context would you need which number? Uh which kind of criteria need to be met. Uh but this is most of the time also not done. So it's just choosing a number between zero and one and um depending on the uh context it might just uh totally change perspective and is therefore rather low signal and also probably inconsistent um across runs. What we see uh working instead is uh really looking into um what are the quality criteria that you want to work uh for your application. So what does good mean? Uh so for example instead of correctness the answer is based on the knowledge base yes no. So for example, if there's a snippet about internal information in the answer and you can check is it actually also in the ret retrieved context we did in the previous step or also if you're frequently struggling with cor using your correct brand voice you could look into okay um is it this time correct? Did we make sure that our name was written correctly? I've also seen companies that have English names that are checking for did we not accidentally translate our name to Spanish? things like this. And also um uh one thing that works well because it's also this categorization yes no is maybe some known failure modes. So which out of these five types of failures uh happened here and um can you categorize it? And these are only a few examples. Um we usually see that these are created by looking into the data, understanding what good means, what are things that are important to you and especially to your domain experts and creating these high signal um feedback loops. Um because if you can't do code compiles, you need to wrap your head a little bit differently around um what is good and what is not. So um this is the one side. The other part is of course um the volume that you need. We had a like luckily we could have used thousands of examples. there's so many papers out there. We opted for like 200 train 100 validate um 300 test um just because we wanted to see that it properly generalizes um but we made the experience that this already got gave back a lot of um high signal feedback um so depending on the complexity of your application and the steps in there of course it might differ um but having the volume there and having the high signal feedback um oops are the foundation. So how do you get there? So the most important thing we see is working with your experts and this is like repeated advice and everyone's saying it and everyone's saying talk to your customers. Um but here it's something you can actually start to encode the domain expertise into very concrete examples. So use them to create examples what should come out of it. Look at those samples run sample runs together with them and also ask okay but why is it like this here and here um the other way? Uh because like this you can often also get information out of them that they implicitly think is clear and also um really understand um how the decisions are taken there and like this identify failure modes of your application and also define what good means because you probably need both and then use them to find these high signal evaluators. Then um as soon as you have this baseline and um you might hit production um review this data and don't review it only with your coding agents but review it as a human. look through it, understand like what is the the scope initially and um uh what is then being done with this application. This can shift over time. The failure modes can change. Um people might try to do other things and it's potential for feature um expansion and to really understand how the system works and fails in production um is necessary to to cover the typical failure modes and catch them uh before your customers do. And then um finally um think about the whole thing as a system that generalizes. So you're trying to build up examples that are representative of what you might want to hit uh in production at some point. And um for this to properly generalize, you need to uh bake in mechanisms for validation. Uh so like in traditional machine learning context, um the validation approach is actually quite typical. So people um all the time make sure that they're actually validating if uh things don't overfitit. And um this combined with like a real mechanism combined with like the instructions in your loop and giving the system an escape hatch instead of having it work for hours and hours uh hitting a wall and uh burning tokens. Um yeah are very important divas um to actually uh make sure that you're not um just burning tokens but actually looping towards um a system where a human and your agent together collaborate on an improving system. Perfect. Thank you so much for uh listening to me. Um very excited that you took the time. And if you want to know more about Lenfuse and also how we think about these topics, you can find more on length.com. And I'm super excited to meet some of you at some point in person.