--- schema: 1 type: video title: 'The Self-Improving OSS Agent Stack — Marc Klingen, Langfuse' description: >- Improving an agent used to mean a human reading traces every week. Now agents can do most of that loop for you. Marc Klingen, co-founder of Langfuse, shows how teams are moving from manually improving their agents to letting AI run the improvement loop. He explains the reference stack that emerged for building agents, which connects online tracing and monitoring with offline datasets, experiments and evals, and maps the layers of loops from next-token prediction up to human goal-setting. AI is already good at proposing fixes, and teams are starting to use it to maintain datasets and eval criteria too, while humans stay in charge of direction. In a demo, a coding agent uses Langfuse's agent skills to find that Langfuse's own changelog writer leaks internal jargon, updates the evals, and back-tests a better version before shipping it. In this talk: • The online + offline loop: traces, datasets, experiments and evals • Where humans still need to stay in the loop, and where AI can take over • Using implicit user signals like edits and approvals as feedback • Why owning your trace data matters when agents read far more of it SPEAKER Marc Klingen, Co-founder, Langfuse LinkedIn: https://www.linkedin.com/in/marcklingen/ X: https://x.com/marcklingen GitHub: https://github.com/marcklingen Website: https://marcklingen.com LINKS Langfuse: https://langfuse.com Langfuse on GitHub: https://github.com/langfuse/langfuse Langfuse on X: https://x.com/langfuse CHAPTERS 0:00 Intro 0:42 Loops, not prompts 1:12 How far models have come 1:42 The online + offline agent loop 3:06 From manual to automated 3:16 Layers of loops 4:06 Where humans still matter 5:01 AI proposing fixes 5:46 AI maintaining datasets and evals 7:00 The target keeps moving 8:10 Time invested vs quality 10:04 Feeding in user signals 10:24 Demo: a self-improving changelog writer 12:54 Demo: catching leaked jargon 13:53 Demo: back-testing V1 vs V2 15:13 The pattern the best teams use 15:58 Own your data layer Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #Langfuse #LLMEvals #AIEngineer canonical: 'https://liquidgold.cc/watch/TeErpYBUIeM' markdown: 'https://liquidgold.cc/watch/TeErpYBUIeM.md' llms: 'https://liquidgold.cc/watch/TeErpYBUIeM/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:19:46.624Z' language: null children: [] videoId: TeErpYBUIeM durationSeconds: 1027 publishedAt: '2026-10-06T17:00:19.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 114 endSeconds: 190 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=114' heldReason: rights - ordinal: 1 text: null startSeconds: 114 endSeconds: 190 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=114' heldReason: rights - ordinal: 2 text: null startSeconds: 211 endSeconds: 255 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=211' heldReason: rights - ordinal: 3 text: null startSeconds: 310 endSeconds: 322 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=310' heldReason: rights - ordinal: 4 text: null startSeconds: 267 endSeconds: 310 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=267' heldReason: rights - ordinal: 5 text: null startSeconds: 353 endSeconds: 365 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=353' heldReason: rights - ordinal: 6 text: null startSeconds: 365 endSeconds: 423 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=365' heldReason: rights - ordinal: 7 text: null startSeconds: 365 endSeconds: 423 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=365' heldReason: rights - ordinal: 8 text: null startSeconds: 434 endSeconds: 499 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=434' heldReason: rights - ordinal: 9 text: null startSeconds: 510 endSeconds: 606 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=510' heldReason: rights - ordinal: 10 text: null startSeconds: 606 endSeconds: 617 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=606' heldReason: rights - ordinal: 11 text: null startSeconds: 640 endSeconds: 774 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=640' heldReason: rights - ordinal: 12 text: null startSeconds: 785 endSeconds: 840 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=785' heldReason: rights - ordinal: 13 text: null startSeconds: 851 endSeconds: 917 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=851' heldReason: rights - ordinal: 14 text: null startSeconds: 917 endSeconds: 927 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=917' heldReason: rights - ordinal: 15 text: null startSeconds: 970 endSeconds: 1027 watch: 'https://liquidgold.cc/watch/TeErpYBUIeM?t=970' heldReason: rights --- # The Self-Improving OSS Agent Stack — Marc Klingen, Langfuse Improving an agent used to mean a human reading traces every week. Now agents can do most of that loop for you. Marc Klingen, co-founder of Langfuse, shows how teams are moving from manually improving their agents to letting AI run the improvement loop. He explains the reference stack that emerged for building agents, which connects online tracing and monitoring with offline datasets, experiments and evals, and maps the layers of loops from next-token prediction up to human goal-setting. AI is already good at proposing fixes, and teams are starting to use it to maintain datasets and eval criteria too, while humans stay in charge of direction. In a demo, a coding agent uses Langfuse's agent skills to find that Langfuse's own changelog writer leaks internal jargon, updates the evals, and back-tests a better version before shipping it. In this talk: • The online + offline loop: traces, datasets, experiments and evals • Where humans still need to stay in the loop, and where AI can take over • Using implicit user signals like edits and approvals as feedback • Why owning your trace data matters when agents read far more of it SPEAKER Marc Klingen, Co-founder, Langfuse LinkedIn: https://www.linkedin.com/in/marcklingen/ X: https://x.com/marcklingen GitHub: https://github.com/marcklingen Website: https://marcklingen.com LINKS Langfuse: https://langfuse.com Langfuse on GitHub: https://github.com/langfuse/langfuse Langfuse on X: https://x.com/langfuse CHAPTERS 0:00 Intro 0:42 Loops, not prompts 1:12 How far models have come 1:42 The online + offline agent loop 3:06 From manual to automated 3:16 Layers of loops 4:06 Where humans still matter 5:01 AI proposing fixes 5:46 AI maintaining datasets and evals 7:00 The target keeps moving 8:10 Time invested vs quality 10:04 Feeding in user signals 10:24 Demo: a self-improving changelog writer 12:54 Demo: catching leaked jargon 13:53 Demo: back-testing V1 vs V2 15:13 The pattern the best teams use 15:58 Own your data layer Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #Langfuse #LLMEvals #AIEngineer Canonical: https://liquidgold.cc/watch/TeErpYBUIeM · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.