--- schema: 1 type: video title: >- Everything You Need To Know About Agent Observability — Danny Gollapalli & Zubin Koticha, Raindrop description: >- Agent failures do not look like normal software failures. In this workshop, the Raindrop team breaks down what it actually takes to monitor production agents, from explicit signals like tool errors, latency, and cost to fuzzier signals like user frustration, refusals, task failure, and capability gaps. The session covers how to move beyond evals toward real production observability, how to use classifiers, regex, and experiments to catch regressions, and how to instrument self-diagnostics so agents can report their own failures and strange behavior. If you're running agents in production, this is a practical framework for understanding what is going wrong and how to catch it early. Speaker info: - https://x.com/benhylak - https://www.linkedin.com/in/zkoticha - https://www.linkedin.com/in/joseph-daniel-gollapalli-a371a4138/ Timestamps 0:14 Introduction and the problem of agent failures 1:48 Moving from evals to production monitoring 3:33 The two types of signals: explicit and implicit 4:47 Using classifier signals for observability 6:38 Leveraging regex for signal detection 7:30 Using experiments to validate improvements 9:42 Q&A session: Statistical relevance and experimental design 16:07 Introduction to self-diagnostics 20:15 Workshop: Coding agent demonstration 24:01 Live demo: Triggering and handling tool failure 30:26 Best practices for self-diagnostic implementation 32:20 Q&A: Real-world use cases and triage 40:02 Q&A: Managing fast-paced experimentation 44:21 Q&A: Trace visualization and data export --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/-aM2EDTiaMs-everything-you-need-to-know-about-agent-observability Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/-aM2EDTiaMs' markdown: 'https://liquidgold.cc/watch/-aM2EDTiaMs.md' llms: 'https://liquidgold.cc/watch/-aM2EDTiaMs/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T14:12:15.266Z' language: null children: [] videoId: '-aM2EDTiaMs' durationSeconds: 3025 publishedAt: '2026-05-07T13:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 26 endSeconds: 108 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=26' heldReason: rights - ordinal: 1 text: null startSeconds: 26 endSeconds: 108 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=26' heldReason: rights - ordinal: 2 text: null startSeconds: 119 endSeconds: 252 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=119' heldReason: rights - ordinal: 3 text: null startSeconds: 119 endSeconds: 252 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=119' heldReason: rights - ordinal: 4 text: null startSeconds: 119 endSeconds: 252 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=119' heldReason: rights - ordinal: 5 text: null startSeconds: 252 endSeconds: 262 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=252' heldReason: rights - ordinal: 6 text: null startSeconds: 262 endSeconds: 352 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=262' heldReason: rights - ordinal: 7 text: null startSeconds: 363 endSeconds: 489 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=363' heldReason: rights - ordinal: 8 text: null startSeconds: 363 endSeconds: 489 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=363' heldReason: rights - ordinal: 9 text: null startSeconds: 499 endSeconds: 629 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=499' heldReason: rights - ordinal: 10 text: null startSeconds: 499 endSeconds: 629 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=499' heldReason: rights - ordinal: 11 text: null startSeconds: 640 endSeconds: 763 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=640' heldReason: rights - ordinal: 12 text: null startSeconds: 763 endSeconds: 774 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=763' heldReason: rights - ordinal: 13 text: null startSeconds: 774 endSeconds: 863 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=774' heldReason: rights - ordinal: 14 text: null startSeconds: 982 endSeconds: 1091 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=982' heldReason: rights - ordinal: 15 text: null startSeconds: 982 endSeconds: 1091 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=982' heldReason: rights - ordinal: 16 text: null startSeconds: 1091 endSeconds: 1102 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=1091' heldReason: rights - ordinal: 17 text: null startSeconds: 1102 endSeconds: 1221 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=1102' heldReason: rights - ordinal: 18 text: null startSeconds: 1323 endSeconds: 1442 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=1323' heldReason: rights - ordinal: 19 text: null startSeconds: 1613 endSeconds: 1704 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=1613' heldReason: rights - ordinal: 20 text: null startSeconds: 1715 endSeconds: 1804 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=1715' heldReason: rights - ordinal: 21 text: null startSeconds: 2014 endSeconds: 2113 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=2014' heldReason: rights - ordinal: 22 text: null startSeconds: 2014 endSeconds: 2113 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=2014' heldReason: rights - ordinal: 23 text: null startSeconds: 2234 endSeconds: 2321 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=2234' heldReason: rights - ordinal: 24 text: null startSeconds: 2331 endSeconds: 2418 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=2331' heldReason: rights - ordinal: 25 text: null startSeconds: 2517 endSeconds: 2528 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=2517' heldReason: rights - ordinal: 26 text: null startSeconds: 2630 endSeconds: 2734 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=2630' heldReason: rights - ordinal: 27 text: null startSeconds: 2744 endSeconds: 2854 watch: 'https://liquidgold.cc/watch/-aM2EDTiaMs?t=2744' heldReason: rights --- # Everything You Need To Know About Agent Observability — Danny Gollapalli & Zubin Koticha, Raindrop Agent failures do not look like normal software failures. In this workshop, the Raindrop team breaks down what it actually takes to monitor production agents, from explicit signals like tool errors, latency, and cost to fuzzier signals like user frustration, refusals, task failure, and capability gaps. The session covers how to move beyond evals toward real production observability, how to use classifiers, regex, and experiments to catch regressions, and how to instrument self-diagnostics so agents can report their own failures and strange behavior. If you're running agents in production, this is a practical framework for understanding what is going wrong and how to catch it early. Speaker info: - https://x.com/benhylak - https://www.linkedin.com/in/zkoticha - https://www.linkedin.com/in/joseph-daniel-gollapalli-a371a4138/ Timestamps 0:14 Introduction and the problem of agent failures 1:48 Moving from evals to production monitoring 3:33 The two types of signals: explicit and implicit 4:47 Using classifier signals for observability 6:38 Leveraging regex for signal detection 7:30 Using experiments to validate improvements 9:42 Q\&A session: Statistical relevance and experimental design 16:07 Introduction to self-diagnostics 20:15 Workshop: Coding agent demonstration 24:01 Live demo: Triggering and handling tool failure 30:26 Best practices for self-diagnostic implementation 32:20 Q\&A: Real-world use cases and triage 40:02 Q\&A: Managing fast-paced experimentation 44:21 Q\&A: Trace visualization and data export --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/-aM2EDTiaMs-everything-you-need-to-know-about-agent-observability Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/-aM2EDTiaMs · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.