--- schema: 1 type: video title: >- Mind the Gap (In your Agent Observability) — Amy Boyd & Nitya Narasimhan, Microsoft description: >- Agents drift. Models change, prompts get tweaked, edge cases accumulate, and the gap between what your agent does and what you need it to do widens without you noticing. Amy and Nitya walk through Microsoft Foundry's observability stack: tracing built on OpenTelemetry, built-in evaluators for quality, safety, and agentic metrics like intent resolution and task adherence, and red teaming where a second AI attacks your agent with adversarial prompts to find vulnerabilities before your users do. The piece worth watching for is the observe skill demo. You point it at an agent with no eval dataset, no baselines, nothing. It generates the dataset, runs batch evaluations, optimizes the prompt, compares versions, and rolls back to the best one... all from a single prompt to a coding agent. The skill shows its reasoning at each step, which is where the real value is: it surfaces the failures you didn't know to look for. Speaker info: - https://x.com/NityaNarasimhan - https://www.linkedin.com/in/nityan/ - https://x.com/AmyKateNicho - https://www.linkedin.com/in/amykatenicho/ --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/iOXM3zE-2dk-mind-the-gap-in-your-agent-observability Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/iOXM3zE-2dk' markdown: 'https://liquidgold.cc/watch/iOXM3zE-2dk.md' llms: 'https://liquidgold.cc/watch/iOXM3zE-2dk/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T14:11:12.808Z' language: null children: [] videoId: iOXM3zE-2dk durationSeconds: 4807 publishedAt: '2026-05-14T16:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 114 endSeconds: 234 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=114' heldReason: rights - ordinal: 1 text: null startSeconds: 246 endSeconds: 355 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=246' heldReason: rights - ordinal: 2 text: null startSeconds: 365 endSeconds: 469 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=365' heldReason: rights - ordinal: 3 text: null startSeconds: 480 endSeconds: 592 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=480' heldReason: rights - ordinal: 4 text: null startSeconds: 731 endSeconds: 741 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=731' heldReason: rights - ordinal: 5 text: null startSeconds: 885 endSeconds: 983 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=885' heldReason: rights - ordinal: 6 text: null startSeconds: 994 endSeconds: 1104 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=994' heldReason: rights - ordinal: 7 text: null startSeconds: 1338 endSeconds: 1349 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=1338' heldReason: rights - ordinal: 8 text: null startSeconds: 1807 endSeconds: 1819 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=1807' heldReason: rights - ordinal: 9 text: null startSeconds: 2039 endSeconds: 2136 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=2039' heldReason: rights - ordinal: 10 text: null startSeconds: 2385 endSeconds: 2395 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=2385' heldReason: rights - ordinal: 11 text: null startSeconds: 2797 endSeconds: 2919 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=2797' heldReason: rights - ordinal: 12 text: null startSeconds: 2930 endSeconds: 3051 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=2930' heldReason: rights - ordinal: 13 text: null startSeconds: 3061 endSeconds: 3179 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=3061' heldReason: rights - ordinal: 14 text: null startSeconds: 3300 endSeconds: 3312 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=3300' heldReason: rights - ordinal: 15 text: null startSeconds: 3535 endSeconds: 3546 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=3535' heldReason: rights - ordinal: 16 text: null startSeconds: 3645 endSeconds: 3756 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=3645' heldReason: rights - ordinal: 17 text: null startSeconds: 3767 endSeconds: 3856 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=3767' heldReason: rights - ordinal: 18 text: null startSeconds: 3948 endSeconds: 3958 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=3948' heldReason: rights - ordinal: 19 text: null startSeconds: 4058 endSeconds: 4158 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=4058' heldReason: rights - ordinal: 20 text: null startSeconds: 4501 endSeconds: 4621 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=4501' heldReason: rights - ordinal: 21 text: null startSeconds: 4632 endSeconds: 4744 watch: 'https://liquidgold.cc/watch/iOXM3zE-2dk?t=4632' heldReason: rights --- # Mind the Gap (In your Agent Observability) — Amy Boyd & Nitya Narasimhan, Microsoft Agents drift. Models change, prompts get tweaked, edge cases accumulate, and the gap between what your agent does and what you need it to do widens without you noticing. Amy and Nitya walk through Microsoft Foundry's observability stack: tracing built on OpenTelemetry, built-in evaluators for quality, safety, and agentic metrics like intent resolution and task adherence, and red teaming where a second AI attacks your agent with adversarial prompts to find vulnerabilities before your users do. The piece worth watching for is the observe skill demo. You point it at an agent with no eval dataset, no baselines, nothing. It generates the dataset, runs batch evaluations, optimizes the prompt, compares versions, and rolls back to the best one... all from a single prompt to a coding agent. The skill shows its reasoning at each step, which is where the real value is: it surfaces the failures you didn't know to look for. Speaker info: - https://x.com/NityaNarasimhan - https://www.linkedin.com/in/nityan/ - https://x.com/AmyKateNicho - https://www.linkedin.com/in/amykatenicho/ --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/iOXM3zE-2dk-mind-the-gap-in-your-agent-observability Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/iOXM3zE-2dk · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.