Talk by Marc Klingen (co-founder of Langfuse) on moving from manually improving AI agents to letting AI run parts of the improvement loop. He presents a reference stack connecting online tracing and monitoring with offline datasets, experiments, and evals, and maps layers of loops from next-token prediction up to human goal-setting. Topics include where AI can propose fixes and maintain datasets and eval criteria while humans stay in charge of direction, using implicit user signals (edits, approvals) as feedback, and why owning trace data matters. Includes a demo where a coding agent uses Langfuse agent skills to detect jargon leakage in a changelog writer, update evals, and back-test an improved version. Recorded at AI Engineer World's Fair 2026, San Francisco. # Video Identity Observations ## Video Overview - **Title**: "The Self-Improving OSS Agent Stack — Marc Klingen, Langfuse" - **Format**: Conference talk (speaker notes "super excited to be here," references "our booth" and post-talk discussion) - **Presenter self-identification**: Marc Klingen, co-founder of Langfuse ## Explicit Self-Description - **Role**: "I'm Mark, one of the founders of Langfuse." - **Stated perspective basis**: "my view is based on working with our community of people that use Langfuse to trace and evaluate their agents" — claims authority derived from community/customer observation rather than pure theory. - **Product origin story**: "we started working on Langfuse… right when ChatGPT was released," including an early experiment: "we built like GitHub issue to pull request automation similar to what Devin or Cursor are building." - **Vendor framing acknowledged**: He attempts neutrality — "I don't want to name competitors" and "I try to keep the talk mostly to more generic takeaways that work with whatever stacks you're using" — while noting "Langfuse is like well positioned for this." ## Claimed Scope - Topic: how teams move from **manually driving** the agent-improvement loop to **AI-driven self-improvement**, and the "open source reference stack" emerging for it. - Conceptual framework offered: layered loops (token loop → reasoning loops → human goal-setting), the online/offline loop of tracing, evals, datasets, and the "promised land" of low time investment with human direction-setting. - Two industry trends predicted: AI observability systems becoming **read-intensive** ("agents can chew through so much more data") and the need to **own the data layer** ("you don't want them locked up in a more commercial system… where you need to sample data"). ## Intended Audience - **AI/ML engineering teams building agent applications**, evidenced by: - Assuming familiarity with tooling: "ML ops toolings like MLflow, Weights and Biases," GitHub Copilot, "Codex goal mode." - Practitioner pain-point framing: "it's really tedious to get things going," teams asking "how can we take ourselves out of the loop to automate it." - Offer of follow-up: "happy to give you a more deep dive demo at our booth or after this talk." ## Observable Presentation Patterns - **Demo-driven**: presents a live demo of a "changelog writer" agent — a Langfuse internal use case — showing trace analysis, dataset/evaluator updates, and A/B backtesting ("v1 baseline" vs. "v2 candidate" with "positive delta"). - **Educational positioning**: "since starting working on Langfuse we try to educate teams how to do this process well." - **Balanced-claim style**: flags risks of full automation ("you risk that you just create more slop"; overfitting to datasets) alongside benefits. - **Content marketing element**: references "a blog post of ours" and closes with commercial availability ("we've worked with many customers on this"). ## Claims vs. Demonstrated - **Creator claims**: Langfuse is "well positioned" for self-improving agent loops; best users "already" collect production signals and run agents on loops (via cron "every day, every week"). - **Demonstrated**: one internal demo application; general workflow patterns (approvals/change requests as feedback signal, human review of dataset/evaluator changes). No external customer data is shown in the transcript. # Delivery Observations: Marc Klingen, "The Self-Improving OSS Agent Stack" > **Scope:** This document describes observable delivery features in the transcript only: vocabulary, recurring phrases, fillers, pacing markers, and teaching moves. It makes no claims about the speaker's personality or motivations. The transcript is automatic and contains recognition artifacts (for example, "Mark" for the title's "Marc," and "language" where the context suggests "Langfuse"), so some patterns may reflect transcription choices rather than delivery. --- ## 1. Core Vocabulary ### 1.1 The dominant term: "loop" The word **"loop"** and its variants appear in almost every section and anchor the talk's framing. - > "I think this is the year uh where uh teams talk about like upleveling the the level of uh where where they operate themselves" - > "this kind of loop where traditionally I mean you have like obsibility and the data docks here" - > "you can use agents for that" (used in the context of the offline loop) - > "So what we see how um we go through them one by one" The speaker also uses **"in the loop"** and **"out of the loop"** as a contrast for human involvement: - > "it's usually like the ask them like oh models got better now how can we take ourselves out of the loop" - > "you want to be in that loop involved because you set there by the direction" ### 1.2 Level and layering language The speaker frames automation as stacked levels: - > "on the very high level it's uh like like the most abstract form of thinking where you as a human are still involved" - > "then level three was possible like more in 2024 and now2526 is more like the higher level loops" Related recurring terms: **"high level,"** **"lowest level,"** **"higher level loops,"** and **"lower level loops."** ### 1.3 Domain terms (used consistently) | Term | Example from transcript | |---|---| | **tracing / traces** | "I'm Mark, one of the founders of Langfuse" is followed by "use language to trace and evaluate their their agents" | | **evals / evaluators** | "new evaluators" and "evil criteria" (ASR for "eval criteria") | | **data sets** | "align data sets of example questions of how the agents are used" | | **hill climb** | "then this how do we hill climb against the data sets?" | | **back test** | "we can back test it basically on the updated data set" | | **baseline / candidate** | "the v1 baseline" and "the v2 candidate" | | **slop** | "you just create more slop" | ### 1.4 Informal and colloquial vocabulary - **"yolo"**: "this is a bit I mean internal application it's a bit yolo" - **"sabotage"**: "we kind of like sabotage our releases" - **"chew through"**: "agents can like like chew through so much more data" - **"tedious"**: "it's because it's really tedious" --- ## 2. Recurring Phrases and Hedges ### 2.1 Self-framing hedges The speaker often qualifies claims with personal framing: - > "obviously my view is based on working with our community of people that use language to trace and evaluate their their agents" - > "I mean that's what teams have been for example using length use for for the last years" - > "I'd say" (appears repeatedly, e.g., "Over the last I'd say month") ### 2.2 Scope-setting statements - > "I try to keep the talk mostly to to like more generic takeaways that work with whatever stacks you're using." - > "I don't want to name competitors. I want to be concise." - > "happy to chat about the different pros and cons after the talk." - > "happy to talk one about uh oneonone about this if you're interested" ### 2.3 Mid-talk check-ins and pauses - > "I'll pause a couple times because otherwise it it'll go it'll go by very fast." - > "I'll stop here again where we have basically the baseline." These are explicit delivery-pacing signals and appear at least twice. ### 2.4 Summary phrases - > "So what we see..." / "So what we have seen..." - > "this is the high level view of how I would conceptualize..." - > "that's very high level what we've seen" - > "just wanted to to wrap it up very concisely" --- ## 3. Filler and Disfluency Patterns Fillers are frequent throughout, and the speaker often repeats words during restarts. ### 3.1 Common fillers - **"uh" / "um"**: appear dense in the opening and throughout - **"like"**: used as a hedge or connective ("like a link of like tracing") - **"I mean"**: frequent connective ("I mean obviously my view...") - **"kind of like"**: frequent softener ("it's kind of like just like a a problem") - **"basically"**: frequent summarizer ("basically reproduce the issue") ### 3.2 Repetition and restarts - > "the the level of uh where where they operate" - > "it it'll go it'll go by very fast" - > "good good models" - > "I mean that's what teams have been" ### 3.3 Opening disfluency The opening contains an unscripted moment: - > "Everyone, super excited to be here. Ah, okay. My laptop was alerting me that this starts now." This is the only clear meta-comment on the setting. --- ## 4. Sentence Structure and Rhythm - **Long, run-on sentences** are common, often linked by "and," "but," or "so," with clauses added after the main point: > "So uh like the the change writer..." (paraphrased pattern; see section 5 for full examples) - **Short emphatic sentences** appear occasionally for contrast: > "We don't need to over fit our application to this random quirk that we found in production, but we want to like be in the loop to to monitor this." > "This is so like where where humans need to review." - **Hedged generalizations** precede claims: > "I think you want to be in this kind of like promised land..." --- ## 5. Teaching Habits ### 5.1 Concrete examples before abstractions The speaker repeatedly grounds ideas in specific use cases: - A **GitHub issue to pull request** automation, from the early era: > "we built like github issue to pull request automation similar to what devon or kurs are building" - A **customer support agent** used as a running example: > "if we have an customer support application like support agent then the data set should be like common support questions" - A **changelog writer** used as the final worked demo: > "we thought about like putting together like a change lock writer that just uploads like updates a lot of like the public assets based on what we've shipped" ### 5.2 Analogies and metaphors - > "you assemble the plane while you're flying it" - > "this kind of like promised land of you don't really need to invest a lot of time" - > "you pull yourself out of a lot of like the more manual and tedious work" ### 5.3 Explicit contrast framing The speaker often sets two options against each other: - > "you either benchmark on data that's inaccurate ... or you monitoring like production data but you don't really benchmark offline" - > "if you implement this AI workflow well ... you are in this era ... very high time invest" - > "if you fully automate it, I would say you don't really need to invest any time" This "A or B / well done vs. fully automated" structure recurs in the closing sections. ### 5.4 Worked demo narration During the changelog demo, the speaker narrates the agent's reasoning step by step: - > "what we for example here identified that the writer uh so the chain lock rider is factually uh like very correct in how it makes changes to documentation but clarity is low" - > "then the agent suggests changes to the data sets and evaluators" - > "so we are like okay we are good of of just merging this" This narration pattern (identify → propose → test → accept) is explicit and sequential. ### 5.5 Caveats and risk framing The speaker repeatedly names risks alongside recommendations: - > "you have the risk of like overfitting it to um to like the data set and the failure definition" - > "you risk that like it produces a lot of tokens, but don't really make sense" --- ## 6. Structural Signposting The talk is organized with verbal signposts: 1. **Opening / framing:** "what we have seen have people like self-improve agents" 2. **History of models:** "back when we started working on langfuse this was right when ch was released" 3. **Loop model:** "we use the visualization ... how we go through them one by one" 4. **Demo:** "we put together like a super quick demo application" 5. **Closing / implications:** "this really drives the need for like a scalable cheap solution" The speaker explicitly says "one by one" when moving through levels, and "So what we see..." when summarizing. --- ## 7. Observed Limitations and Uncertainty - **Transcript quality:** Several terms are clearly ASR errors ("ch" for ChatGPT, "evil" for evals, "length" for Langfuse, "obsibility" for observability). Vocabulary counts and exact phrasing may not match the spoken delivery. - **Repetition evidence:** The "loop" pattern is strongly repeated. Other patterns, such as "I'd say," "basically," and the pause signals, appear multiple times but less densely. - **Speaking pace:** Pace cannot be measured from the transcript alone. The speaker's own pause cues ("I'll pause a couple times") suggest the pace was deliberately slowed in places, but this is a self-report, not observed timing. - **Audience interaction:** The transcript contains no audience questions or responses, so interactive delivery habits cannot be assessed. - **Ending:** The talk ends with a direct invitation for one-on-one follow-up, which is a repeated pattern ("happy to chat / happy to talk") but appears only in the opening and closing. # Position Record: "The Self-Improving OSS Agent Stack" — Marc Klingen, Langfuse ## Speaker **Marc Klingen**, co-founder of Langfuse (per video title and self-introduction: "I'm Mark, one of the founders of Langfuse"). He notes his view is grounded in the Langfuse user community but frames the talk as generic takeaways. No other speakers appear. ## Core Thesis Teams are moving from **manually driving the agent-improvement loop** to using **AI to drive that loop**, while keeping humans involved at the highest-level decision points. Klingen calls the optimal state a "**promised land**" of low time investment with high agent quality. ## Key Claims and Reasoning ### 1. Agent-building requires a combined online/offline loop - Online: tracing and monitoring how users actually use the agent. - Offline: datasets, experimentation, offline evaluation, deployment, learning from production again. - **Reasoning**: keeping these separate means either "you benchmark on data that's inaccurate... not in sync with what's happening in production, or you're monitoring production data but you don't really benchmark offline." - He states this new category (beyond observability data tools and MLOps tools like MLflow/W&B) is why Langfuse was built. ### 2. Higher-level AI loops became possible because models improved - Timeline (as he presents it): Langfuse started when ChatGPT released; back then even GitHub-issue-to-PR automation only worked on **single HTML files** because multi-file edits were too complicated. 2024 enabled "level three"; 2025–26 enables higher loops where agents "reason about what to even fix... how to verify whether they actually work." ### 3. Humans must stay in the loop at specific points He identifies remaining human involvement in: - **Aligning datasets** of example questions (reviewing what gets amended). - **Reviewing proposed fixes** — because of the risk of **overfitting** to the dataset and failure definition; agents find failure patterns that may not matter ("we don't need to overfit our application to this random quirk"). - Later: reviewing AI-maintained **eval criteria and datasets**, since these "create the new boundary for how then other agents try to auto improve against it" — otherwise "you risk that you just create more slop." ### 4. Where AI is used most today - **Proposing fixes**: hill-climbing against datasets/eval criteria while maintaining a fixed optimization boundary. - **Newer (~last month, per his observation)**: AI maintaining eval criteria and datasets — e.g., keeping a customer-support dataset aligned with actual user behavior, proposing new evaluators (conciseness, matching user language, no competitor mentions). ### 5. The "moving target" argument > The target of an AI application is always changing... "you assemble the plane while you're flying it." - High-level goals ("automate customer support") are vague; error cases reveal what actually needs doing along the way. ### 6. Three investment/quality scenarios (his conceptualization) - **Manual loop well-implemented**: high time investment, high quality. - **Fully automated**: near-zero time but "you risk that it produces a lot of tokens, but don't really make sense." - **Promised land**: human sets direction (goals, dataset/evaluator changes); AI automates tedious lower loops. Quality "can be even higher" because AI reviews more error cases than human patience allows — manual work is "bottlenecked by your own time... perseverance." ### 7. Implicit production signals as free input Customer complaints, corrections, or proposed-message acceptance/rejection feed the improvement loop without manual developer effort. ## Demo Claims (Langfuse internal "changelog writer" agent) - **Problem**: engineering ships so fast with AI that communicating releases (docs, changelog, socials) becomes the bottleneck; previously engineers handled this for monthly releases. - **Risk stated**: "if we communicate badly about what we've released then... we kind of sabotage our releases." - **Setup**: merge PR → change writer drafts docs/changelog → GitHub PR review (approve or request changes) → writer iterates; approvals/requests feed the eval loop. - **Diagnosed issues**: factually correct but **low clarity**; high human edit ratio; **internal jargon leaking** to customers (e.g., "ingestion pipeline," "batched eval queue" vs. user-domain language). - Agent then proposed dataset/evaluator changes, a v2 implementation, and **back-tested**: same performance on format compliance and accuracy, improved on user-facing language — "a positive delta." - **Qualification**: he calls this "a bit yolo" for merging automatically — acceptable because "this agent never publishes anything to actual production. It just raises PRs on a repo." ## Ecosystem Implications (his closing claims) - **Scalable data systems needed**: agents looping on data produce many queries; Langfuse shifted from **write-intensive** (ingesting traces/evals) to **read-intensive** (agents querying data). - **Own the data layer**: year-old traces are valuable context; avoid commercial lock-in, sampling, and short retention — "retain them for a long time period and not sample them." ## Qualifications and Tensions Preserved - He repeatedly qualifies views as based on **Langfuse's community**, not universal data ("over the last I'd say month, we see more and more teams..."). - The pro-automation argument (AI reviews more, quality higher) coexists with repeated warnings about **slop and overfitting** if human review is removed — an unresolved tension he presents as a design tradeoff rather than resolving. - Demo results ("sounds like a good change") are framed as indicative back-test deltas, with merged changes justified by the narrow non-production blast radius. # Speaker & Provenance Observation Record ## Video Identification - **Title:** The Self-Improving OSS Agent Stack — Marc Klingen, Langfuse - **Video ID:** `01a15ca6-ea4c-78da-88b6-12c5d5f74363` ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | Marc (Mark) Klingen | **Creator / Presenter** | Self-identified: "I'm Mark, one of the founders of Langfuse" | - Single-speaker talk; no guests, narrators, or other identified voices appear in the transcript. - Speaker note: transcript renders his name both as "Mark" and "Marc" (title metadata says "Marc"). ## Attributed Third-Party Claims & References - **"I don't write prompts, I have loops"** — attributed to an unnamed writer ("u Peter writing about it"); speaker references this as an external viewpoint. - **"Auto research becoming really popular"** — general claim, no named source. - **Loop visualization concept** — attributed to "the loopcraft article or like many many articles in the space." - **Devon / Kurs (transcript spelling)** — referenced as building GitHub issue-to-PR automation, similar to a demo the speaker's team previously built. - **GitHub Copilot** — cited as an example of the "lowest level" AI assistance (~2023). - **2023 timing** — speaker states Langfuse work began "right when ch was released" (transcript garbled; likely ChatGPT, but not explicit). ## Referenced Tools & Systems (mentioned, not endorsed as sources) - **Speaker's own product (Langfuse)** — tracing, evals, datasets, prompt management, annotations; heavily demonstrated throughout. - Third-party tools named: **MLflow, Weights & Biases, Codex ("goal mode" implied), X (Twitter)**. ## Promotional Segments — Flagged 1. **Langfuse positioning:** "that's also why we built length fuse" — product-motivation framing throughout the talk. 2. **Demo application:** the changelog-writer demo is built on Langfuse ("length use agent skills but... Langfuse is like well positioned for this"). 3. **Closing promotion (explicit):** - "happy to give you like a more deep dive demo at our at our booth or after this talk" - "we've worked with many customers on this so happy to talk one-on-one about this" — data-layer/scalability pitch tied to Langfuse. ## Provenance Notes & Uncertainty - Transcript is **auto-generated and garbled** in places: "evil criteria" ≈ eval criteria, "length fuse/language/langu" ≈ Langfuse, "chain lock rider/change lock writer" ≈ changelog writer, "ch" likely ChatGPT, "PS" ≈ PRs, "userf facing" ≈ user-facing. - No curated metadata supplied; **comment availability unknown**. - Speaker's observations are explicitly framed as based on **Langfuse's user community** ("my view is based on working with our community") — self-acknowledged potential bias in perspective. - No timestamps available in source. Everyone, super excited to be here. Ah, okay. My laptop was alerting me that this starts now. Uh, hi everyone. I'm Mark, one of the founders of Langfuse. super excited to chat about uh what we have seen how how people like self-improve agents now and what kind of like open source reference stack uh we see emerging. I mean obviously my view is based on working with our community of people that use language to trace and evaluate their their agents but I think I try to keep the talk mostly to to like more generic takeaways that work with whatever stacks you're using. Um but yeah happy to chat about the different pros and cons after the talk. high level. I think this is the year uh where uh teams talk about like upleveling the the level of uh where where they operate themselves where increasingly good good models just abstract everything downstream. I mean coming from I don't write prompts I have loops u Peter writing about it auto research becoming really popular. So I think this is kind of where things are going on the like like producing um application side but also on the how to actually um build good good agent applications and um over the years really models have have expanded a lot in capabilities um back when when we started working on length fuse this was right when ch was released so uh where like even the simplest applications didn't really work so for example we built like github issue to pull request automation similar to what devon or kurs are building but back then we could only manipulate like single page applications like a single HTML file because multifile edits were too complicated. Since then all of these things really accelerated a lot and this really made made loops not possible. At the same time a more like reference tech emerged of how great AI agents are built because building agents is different from building applications. It's like an like a link of like tracing and monitoring like online how users really use an agent application and then like an offline component of how to build data sets how to experiment uh regarding like making changes to these agents and then evaluating offline whether things work then deploying new to production then learning again for real users use the application uh so it's like this this kind of loop where traditionally I mean you have like obsibility and the data docks here in the tracing monitoring side and you have like the ML ops toolings like MF flow weights and biases more on the like offline side where this new category emerged and that's also why we built length fuse because you need to bring the online and the offline together as as otherwise like you either benchmark on data that's inaccurate so the data sets they're not in loop and like uh not in sync with what's happening in production or you monitoring like production data but you don't really benchmark offline so you need to bring the the both together but this has been like a lot of manual labor because you need to look at traces update these data sets think about like new evaluators uh then make these changes create new hypothesis like it's it's like a lot of work going into this process and since starting working on language we try to educate teams how to how to do this process well but it's it's usually like the ask them like oh models got better now how can we take ourselves out of the loop to uh to automate it because it's really tedious uh to get things going and that's what I want to talk about how we see how people go from manually driving this loop to using AI to to drive this loop um we use the visualization um like of for example like the loopcraft article or like many many articles in the space that like layer different loops on top of each other. How I mean at the at the lowest level you just have like this token loop of what's the next what's the next token that's produced and on the very high level it's uh like like the most abstract form of thinking where you as a human are still involved. So what we see how um we go through them one by one high level I just want to highlight these are again possible because models improve. So uh when when we started working on length in 2023 like I mean we had github copilot so like the lowest level or the uh level two u then level three was possible like more in 2024 and now2526 is more like the higher level loops of actually agents reasoning about what what to even fix uh how to propose new fixes and how to how to verify whether they actually work. Where humans are currently still involved is one you need to align data sets of example questions of how the agents are used. This is so like where where humans need to review um like like what what gets amended to the data set and how to then evaluate whether the agents actually work. These are the the two blue things in in loop number three. Then on the proposal of fixes because if um like your agent proposes how to change an agent, you still want to see what was changed because you have the risk of like overfitting it to um to like the data set and the failure definition. Um, so I mean usually agents find all sorts of different uh failure patterns, but maybe some of them don't really matter that much and you're like, huh, this kind of use case for the agent, it doesn't really matter that much. Like this is like out of what the agent should be actually be doing. So we don't need to over fit our application to this random quirk that we found in production, but we want to like be in the loop to to monitor this. And now coming from this loop pattern, we try to dissect it again into more like a workflow where where we currently see AI used the most is in proposing fixes. So if we have production traces and we have like evil criteria and data sets of how to reproduce issues that we find in production, then this how do we hill climb against the data sets? That's like a really cool uh way to kind of like loop your way to a success with with agents because usually they're like 10 different things of what you could be doing. I don't know, use a different model like aggregate context in a new different way. Try whatever you find on X uh in a given week to to see whether it can like uh make a dent into improving the agent. But really like you maintain the boundary of what you optimize against. And you use AI mostly for like proposing improvements to the agent implementation. Like over the last I'd say month, we see more and more teams also using uh like like AI and and their loops to maintain evil criteria and maintain data sets. So uh usually what goes in here is you want to like align how a data set uh looks like for an agent with what actual users are doing. So for example if we have an customer support application like support agent then the data set should be like common support questions but like users do all sorts of things with a support application. So you need to continuously keep it in line with uh with what users are doing and you can use agents for that. And for EVAs as well if you see usual error patterns then you can reproduce the error patterns and propose new evil criteria on these offline data sets. So for example I don't know I want I don't want to name competitors. I want to be concise. I want to answer in the same language as the user actually requested. Um like the question in my customer support application. So you can also propose new evaluators. This is I think rather newish over the last months. um and where we see like the AI more autolooping. However, usually teams that are still involved in reviewing these changes because they create the new boundary for how then other agents try to auto improve against it. If you give this completely out of hand, you risk like that you just create more slop where we've used this in like a blog post of ours where I'd say the target of an AI application is always changing because you don't really know like usually you start with a very high level task of I stick with customer support. You stick with someone says hey in our company customers need to wait a long time for support answers and we have a lot of people employed to respond to them. AI should be able to do this automatically or at least eight in customer support, but it's a very wake target because people don't even know what happens in customer support every day. So, it's kind of like, oh, we want to automate all of it. However, then on the way of doing this, you figure things out based on like error cases of what you even need to do. And that's like this kind of like map that you like you assemble the plane while you're flying it. Um and how we then look at the uh at this kind of like looping graphic is you really want to be like involved in setting the setting like the the goals and how you make changes to data sets evaluators you want to be in that loop involved because you set there by the direction and then AI can automate against it and then off of the newly implemented changes you'll see new error classes and you can then like change cause again on the most highest level but thereby you pull yourself out of a lot of like the more manual and tedious work on the lower levels. Um, this is then the high level view of how how I would conceptualize like the different scenarios that you can find yourself in. I'd say if you implement this uh like AI workflow well of tracing online uh how your agents are working implementing evals on how they are working um having data sets uh of example questions and then benchmarking against them you are in this era in this class like very high time invest because you need a team who really works with this data every week but also quality of the of the agents is pretty high I mean that's what teams have been for example using length use for for the last years and this is how you can get to I'd say success with your agent application. Um, if you fully automate it, I would say you don't really need to invest any time because you're just like, I don't know, codeex goal mode your way to success or like you add higher level loops on top of it, but you also risk that like it produces a lot of tokens, but don't really make sense. So, I think you want to be in this kind of like promised land of you don't really need to invest a lot of time, but you're like in the loop at the most important steps of like setting the direction of where you want to go, but you take yourself out of everything that's tedious. Thereby you're way less time invested than running it manually. But also the quality can be even higher because usually if you need to do everything manually you're bottlenecked by your own time your own like perseverance endurance uh or like patience to look through so much data and AI can just look so through like so much more error cases. This agent will be higher performing and you need less time invest. That's the I think that's what we were going for and how this then looks like is uh you involved at the top but also I mean you want ideally your application to feed in interesting signal. So for example when we talk about the like customer support application use case usually I mean customers can swear at the support agent customers can be like oh no you don't understand uh like this was not correct or like you can propose messages to an internal agent who then can accept them or change them when sending them out. So there's like a lot of like implicit signal that can feed into your process without you as the agent developer needing to do this manually. Uh thereby you have like an like an instream of of signal that your loop can act on. Um so it's either you your team or some signal coming in um and the lower level loops can be automated. That that's high level how we how we think about it. And like uh like we put together like a super quick demo application um that can that can show this. Um so for example like we as the language team we have a big problem now because our like engineering team uses AI a whole lot thus they ship a whole lot and somehow we need to update our customers uh and user base uh what is what is new every week because uh like in recent weeks like a lot was released uh and usually need to update documentation update a change log post about it on socials and historically like engineers did this themselves because they maybe released like a bigger feature like once every month so it's okay to spend a couple of hours. But now if they take themselves out and really ship a lot of like product, they want to ideally uh like automate also that kind of like release process because otherwise we are bottlenecked by how fast we can communicate about changes. So um we thought about like putting together like a change lock writer that just uploads like like updates a lot of like the public assets based on what we've shipped. Um however then the question is is like how we communicate good because if we communicate badly about what we've released then I mean we we kind of like sabotage our releases if we like misrepresent for example what has even been released. So um setup is we have a merge PR, we have a change writer that uh suggests a change to documentation, but then there's like a just like a code review step as like a GitHub PR where someone can review and either like approve on GitHub and get it merged or like submit like change requests of like what needs changes in this um in this in this change in documentation and and the change log. Then the change writer would act on this again to update the draft until it's finally released. And now both of these kinds of like either like approvals or uh requests for changes can get feed into like the AI obserability and eval to to serve as a basis for auto improvement of that change writer agent. So what we've now done is um just use a coding agent pointed at the like length use agent skills but generally I mean this would also work with other systems but yeah length is like well positioned for this to to ask like okay what has happened in this application how can we improve it I'll I'll pause a couple times because otherwise it it'll go it'll go by very fast. So what we for example here identified that the writer uh so the chain lock rider is factually uh like very correct in how it makes changes to documentation but clarity is low. the edit ratio of us humans making ch like submitting change requests is still high and that we leak internal jargon to our customers because maybe our code has like internal comments that we leave there for ourselves for the future but uh like our customers don't think about our application as like I don't know like an ingestion pipeline or as like I don't know some kind of like batched evil Q like they don't know this is like an implementation detail we should talk about it as like an user domain language and that's like uh something we have identified here we're Now the agent suggests changes to the data sets and evaluators to test for leaking jargon and um also like increasing clarity in customer domain language basically. So um agent now closes this gap suggest changes to the data sets and evaluators and now um has an idea of how to change the agent implementation to basically fix this problem because in the end it's kind of like just like a a problem of how we provide context how we like provide the skill that then educates this um content writer and now we can back test it basically on the updated data set to see whether the new implementation would do better than the old implementation and I'll stop here again where we have basically the baseline. So um basically we reproduce the issue of userf facing language. So let's talk about features in a way of how users would talk about it. So we reproduce this kind of problem with the v1 baseline. This basically existing implementation of the agent and agent came up with a new implementation where we see the v2 candidate. We are still doing on format compliance and accuracy in the same way as we did before but we improved on like userf facing language. So we see a positive delta which sounds like a good change now. Um so uh agent reasons about it as this. I mean this is a bit I mean internal application it's a bit yolo. So we are like okay we are good of of just merging this and then acting again on the next change because this agent never publishes anything to actual production. It just raises PS on a repo. So it's okay if it kind of like auto improves and then we just provide feedback again to whatever the next increment is. So um here we use our like prompt management logic so that the agent can just basically feature release this new um implementation and uh now it just summarizes basically um how how we acted on this. So high level um this is what we have seen like most of our best users do already uh when they when they build agents like collect production signals uh track them alongside like detailed execution traces and then run agents like on a loop uh usually like a crown job like every day every week whenever you have like a batch of basically interesting user data again to then act on it. Um either it's just user feedback how how it's happens in this case or internal labeling from like a production system but it could also be you ask people to annotate data like in app for example in langu we also allow like for annotations to feed into that uh into that queue and yeah that's very high level what we've seen happy to give you like a more deep dive um demo at our at our booth or after this talk but just wanted to to wrap it up very concisely with how we see this this space evolves and um I think what's what's interesting here is it drives the need for like a very scalable data system because you'll want to have agents really loop on this data and like produce lots of queries. This we see how langu was historically very right intensive to like um ingest evils and traces. Now it's way more read intensive because agents can like like chew through so much more data. interesting number one and two you want to really like own the data layer because like now the traces of like for example a year ago are interesting context and you don't want to want them to be locked up in like like a more commercial system or something where uh you need to sample data or retain them for a long short time period but we want to like retain them for a long time period and not sample them and yeah this really drives the need for like a scalable cheap solution and we've worked with many customers on this so happy to talk one about uh oneonone about this if you're interested thank you so much for your