--- schema: 1 type: video title: >- Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse description: >- A hands-on workshop on evaluating AI applications where mistakes are expensive. Annabell Schäfer and Doneyli De Jesus from Langfuse use a banking assistant to walk through the whole AI engineering loop. They cover tracing with PII masking, building datasets, turning domain-expert comments into failure modes, LLM-as-judge and code evaluators, and CI that blocks a prompt change when scores drop. Chapters: - 00:00 Why evals are hard - 05:01 The AI engineering loop - 12:47 Demo: a banking assistant with tracing and PII masking - 20:35 Building datasets - 25:18 Expert review and failure modes - 30:07 Choosing evaluators - 35:00 Running evals automatically on every prompt change - 39:44 Dashboards for stakeholders - 45:58 Working with your coding agent - 52:13 Q&A Doneyli De Jesus — https://langfuse.com Annabell Schäfer — https://langfuse.com Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now! canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/evals-for-high-stakes-ai-agents-doneyli-de-jesus-an-9d6e9fb504c5 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/evals-for-high-stakes-ai-agents-doneyli-de-jesus-an-9d6e9fb504c5.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:23:14.046Z' language: null children: [] videoId: obfPzbapMIM durationSeconds: 3212 publishedAt: '2026-10-18T08:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 154 endSeconds: 165 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=154' heldReason: rights - ordinal: 1 text: null startSeconds: 165 endSeconds: 313 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=165' heldReason: rights - ordinal: 2 text: null startSeconds: 313 endSeconds: 323 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=313' heldReason: rights - ordinal: 3 text: null startSeconds: 323 endSeconds: 404 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=323' heldReason: rights - ordinal: 4 text: null startSeconds: 414 endSeconds: 499 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=414' heldReason: rights - ordinal: 5 text: null startSeconds: 509 endSeconds: 630 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=509' heldReason: rights - ordinal: 6 text: null startSeconds: 641 endSeconds: 769 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=641' heldReason: rights - ordinal: 7 text: null startSeconds: 875 endSeconds: 887 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=875' heldReason: rights - ordinal: 8 text: null startSeconds: 986 endSeconds: 1121 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=986' heldReason: rights - ordinal: 9 text: null startSeconds: 1246 endSeconds: 1522 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=1246' heldReason: rights - ordinal: 10 text: null startSeconds: 1831 endSeconds: 2110 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=1831' heldReason: rights - ordinal: 11 text: null startSeconds: 2122 endSeconds: 2393 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=2122' heldReason: rights - ordinal: 12 text: null startSeconds: 2404 endSeconds: 2510 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=2404' heldReason: rights - ordinal: 13 text: null startSeconds: 3154 endSeconds: 3212 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=3154' heldReason: rights - ordinal: 14 text: null startSeconds: 2876 endSeconds: 2965 watch: 'https://liquidgold.cc/watch/obfPzbapMIM?t=2876' heldReason: rights --- # Evals for High-Stakes AI Agents — Doneyli De Jesus & Annabell Schäfer, Langfuse A hands-on workshop on evaluating AI applications where mistakes are expensive. Annabell Schäfer and Doneyli De Jesus from Langfuse use a banking assistant to walk through the whole AI engineering loop. They cover tracing with PII masking, building datasets, turning domain-expert comments into failure modes, LLM-as-judge and code evaluators, and CI that blocks a prompt change when scores drop. Chapters: - 00:00 Why evals are hard - 05:01 The AI engineering loop - 12:47 Demo: a banking assistant with tracing and PII masking - 20:35 Building datasets - 25:18 Expert review and failure modes - 30:07 Choosing evaluators - 35:00 Running evals automatically on every prompt change - 39:44 Dashboards for stakeholders - 45:58 Working with your coding agent - 52:13 Q\&A Doneyli De Jesus — https://langfuse.com Annabell Schäfer — https://langfuse.com Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now! Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/evals-for-high-stakes-ai-agents-doneyli-de-jesus-an-9d6e9fb504c5 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:12 to 2:34) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (2:34 to 5:13) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (5:13 to 6:44) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (6:44 to 8:19) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (8:19 to 10:30) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (10:30 to 12:49) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (12:49 to 14:35) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (14:35 to 16:26) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 8: Document excerpt (16:26 to 18:53) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 9: Document excerpt (18:53 to 20:46) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 10: Document excerpt (20:46 to 25:22) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 11: Document excerpt (25:22 to 30:19) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 12: Document excerpt (30:19 to 35:10) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 13: Document excerpt (35:10 to 39:53) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 14: Document excerpt (39:53 to 41:50) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 15: Document excerpt (41:50 to 43:23) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 16: Document excerpt (43:23 to 46:05) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 17: Document excerpt (46:05 to 47:45) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 18: Document excerpt (47:45 to 49:25) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 19: Document excerpt (49:25 to 50:55) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 20: Document excerpt (50:55 to 52:22) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 21: Document excerpt (52:22 to 53:32) > Held: transcript rights do not permit this chunk's text.