--- schema: 1 type: video title: >- Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse description: >- A frustrated user types in all caps and tells an IT support agent that its answer did not work. Those interactions become signals that a team can find in production traces, rather than anecdotes lost in a chat history. Lotte Verheyden and Annabell Schäfer build that monitoring workflow with Langfuse and a sample support agent called Specs. The application helps a dad with phone questions, and the workshop follows a request through agent observations, model calls and tools to show what a trace needs to contain before it can support useful evaluation. The hands on setup connects a TypeScript application built on the OpenAI Agents SDK to Langfuse. Schäfer adds a judge evaluator for user disagreement, a code evaluator for all caps and a scope check that compares the user's request with the system prompt. Each evaluator targets the relevant messages through explicit variable mappings, so its score answers a defined question. Seeded production traces then give a coding agent enough evidence to investigate patterns through the Langfuse CLI and skill. The analysis surfaces issues such as unnecessary tool calls, retries, latency and incomplete evaluation coverage, and the presenters explore a proposed code change. The closing discussion places this online monitoring inside a continuous improvement loop: turn observed failures into representative datasets, run offline experiments, check regressions and repeat as real usage changes. Speaker info: Annabell Schäfer: - https://www.linkedin.com/in/annabell-schaefer/ - https://github.com/annabellscha Lotte Verheyden: - https://www.linkedin.com/in/lotteverheyden/ - https://github.com/Lotte-Verheyden Workshop resource: - https://langfuse.com/workshop Timestamps: 0:00 - The AI engineering loop 3:08 - Online and offline improvement 5:12 - Build useful traces 6:38 - Monitoring and user feedback 8:01 - Meet Specs, the dad IT support agent 10:06 - Repository and account setup 12:28 - Run the TypeScript application 13:23 - Inspect a live support trace 15:02 - Choose application specific signals 16:21 - Configure the evaluator model 18:16 - Set up a disagreement judge 19:16 - Map trace messages into the evaluator 21:07 - Code based all caps detection 24:16 - Test and execute the monitors 24:58 - Generate frustration and disagreement 26:13 - Inspect evaluator scores 27:47 - Detect out of scope requests 31:51 - Seed production traces 33:50 - Install the CLI and skill 35:24 - Investigate traces with a coding agent 37:24 - Retrieval, retries, and coverage findings 39:05 - Propose a code change 40:03 - Maintain datasets and close the loop 42:25 - Workshop resources and next steps --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/BOCjybvJYM4-youtube-cca48d6d210263c24eef Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/BOCjybvJYM4' markdown: 'https://liquidgold.cc/watch/BOCjybvJYM4.md' llms: 'https://liquidgold.cc/watch/BOCjybvJYM4/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:19:13.087Z' language: null children: [] videoId: BOCjybvJYM4 durationSeconds: 2663 publishedAt: '2026-10-11T18:00:10.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 25 endSeconds: 191 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=25' heldReason: rights - ordinal: 1 text: null startSeconds: 191 endSeconds: 201 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=191' heldReason: rights - ordinal: 2 text: null startSeconds: 201 endSeconds: 316 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=201' heldReason: rights - ordinal: 3 text: null startSeconds: 316 endSeconds: 326 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=316' heldReason: rights - ordinal: 4 text: null startSeconds: 326 endSeconds: 406 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=326' heldReason: rights - ordinal: 5 text: null startSeconds: 406 endSeconds: 418 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=406' heldReason: rights - ordinal: 6 text: null startSeconds: 418 endSeconds: 488 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=418' heldReason: rights - ordinal: 7 text: null startSeconds: 418 endSeconds: 488 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=418' heldReason: rights - ordinal: 8 text: null startSeconds: 499 endSeconds: 608 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=499' heldReason: rights - ordinal: 9 text: null startSeconds: 818 endSeconds: 907 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=818' heldReason: rights - ordinal: 10 text: null startSeconds: 918 endSeconds: 987 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=918' heldReason: rights - ordinal: 11 text: null startSeconds: 998 endSeconds: 1106 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=998' heldReason: rights - ordinal: 12 text: null startSeconds: 1174 endSeconds: 1276 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=1174' heldReason: rights - ordinal: 13 text: null startSeconds: 1276 endSeconds: 1288 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=1276' heldReason: rights - ordinal: 14 text: null startSeconds: 1687 endSeconds: 1915 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=1687' heldReason: rights - ordinal: 15 text: null startSeconds: 1928 endSeconds: 2037 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=1928' heldReason: rights - ordinal: 16 text: null startSeconds: 2047 endSeconds: 2130 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=2047' heldReason: rights - ordinal: 17 text: null startSeconds: 2143 endSeconds: 2253 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=2143' heldReason: rights - ordinal: 18 text: null startSeconds: 2265 endSeconds: 2345 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=2265' heldReason: rights - ordinal: 19 text: null startSeconds: 2419 endSeconds: 2553 watch: 'https://liquidgold.cc/watch/BOCjybvJYM4?t=2419' heldReason: rights --- # Continuously Improving Agents with Langfuse — Annabell Schäfer & Lotte Verheyden, ClickHouse A frustrated user types in all caps and tells an IT support agent that its answer did not work. Those interactions become signals that a team can find in production traces, rather than anecdotes lost in a chat history. Lotte Verheyden and Annabell Schäfer build that monitoring workflow with Langfuse and a sample support agent called Specs. The application helps a dad with phone questions, and the workshop follows a request through agent observations, model calls and tools to show what a trace needs to contain before it can support useful evaluation. The hands on setup connects a TypeScript application built on the OpenAI Agents SDK to Langfuse. Schäfer adds a judge evaluator for user disagreement, a code evaluator for all caps and a scope check that compares the user's request with the system prompt. Each evaluator targets the relevant messages through explicit variable mappings, so its score answers a defined question. Seeded production traces then give a coding agent enough evidence to investigate patterns through the Langfuse CLI and skill. The analysis surfaces issues such as unnecessary tool calls, retries, latency and incomplete evaluation coverage, and the presenters explore a proposed code change. The closing discussion places this online monitoring inside a continuous improvement loop: turn observed failures into representative datasets, run offline experiments, check regressions and repeat as real usage changes. Speaker info: Annabell Schäfer: - https://www.linkedin.com/in/annabell-schaefer/ - https://github.com/annabellscha Lotte Verheyden: - https://www.linkedin.com/in/lotteverheyden/ - https://github.com/Lotte-Verheyden Workshop resource: - https://langfuse.com/workshop Timestamps: 0:00 - The AI engineering loop 3:08 - Online and offline improvement 5:12 - Build useful traces 6:38 - Monitoring and user feedback 8:01 - Meet Specs, the dad IT support agent 10:06 - Repository and account setup 12:28 - Run the TypeScript application 13:23 - Inspect a live support trace 15:02 - Choose application specific signals 16:21 - Configure the evaluator model 18:16 - Set up a disagreement judge 19:16 - Map trace messages into the evaluator 21:07 - Code based all caps detection 24:16 - Test and execute the monitors 24:58 - Generate frustration and disagreement 26:13 - Inspect evaluator scores 27:47 - Detect out of scope requests 31:51 - Seed production traces 33:50 - Install the CLI and skill 35:24 - Investigate traces with a coding agent 37:24 - Retrieval, retries, and coverage findings 39:05 - Propose a code change 40:03 - Maintain datasets and close the loop 42:25 - Workshop resources and next steps --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/BOCjybvJYM4-youtube-cca48d6d210263c24eef Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/BOCjybvJYM4 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.