--- schema: 1 type: video title: 'Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI' description: >- A shopping assistant confidently returns products outside a user's budget because its price filter is broken. Fuad Ali uses the Wonder Toys assistant to show how a production failure can become a complete improvement loop: collect the trace, investigate the code, propose a fix and test it against the cases that failed. The example starts with an application built on the OpenAI Agents SDK and instrumented with Arize. Instead of treating telemetry as a dashboard that a person must inspect, Ali shows how an agent can consume it alongside the application repository and use both to explain what went wrong. Arize Signal groups recurring failures, gathers evidence and opens investigations or pull requests. Arize skills provide a more direct route from a coding agent to traces, datasets and evaluators. Ali then distinguishes a fixed LLM judge rubric from a harness that can inspect tool calls, code and external context while assessing an outcome. The budget example becomes an evaluation that checks whether returned prices actually satisfy the requested range. Agent experiments replay failure datasets against a deployed development endpoint and compare quality, latency and cost before a change ships. Setup pauses and live questions cover keys, tracing and the skills installation. The closing workflow connects automatic detection and repair to validation on real failures, with a reviewable proposal for a human to assess. Speaker info: - https://www.fuadali.com/index.html - https://github.com/Arize-ai/arize-skills - https://github.com/Arize-ai/project-rosetta-stone Timestamps: 0:00 - Self learning loops and telemetry 2:16 - Agents consuming observability 5:02 - A production budget failure 6:09 - The Wonder Toys assistant 7:29 - Repository and tracing setup 8:25 - Investigate the price filter bug 9:19 - Signal and continuous investigation 10:57 - Skills for coding agents 12:37 - Continuous and on demand workflows 13:30 - Workshop setup and keys 19:53 - Inspect the application traces 20:34 - Signal failure clusters and evidence 23:18 - Detect, investigate, and fix 24:45 - Connect the improvement loop 26:34 - Harness as a judge 28:36 - Tool access and external context 29:48 - Evaluate the budget constraint 30:27 - Agent experiments 31:09 - Build and replay failure datasets 32:46 - Compare quality, latency, and cost 33:13 - An autonomous validation loop 35:30 - Instrumentation and skills setup 37:49 - Wrap up --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/fnuNVFgLjHA-youtube-353021d2046547fa53a9 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/building-self-learning-loops-for-your-agent-fuad-al-1f99c2b1e2a1 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/building-self-learning-loops-for-your-agent-fuad-al-1f99c2b1e2a1.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:19:15.192Z' language: null children: [] videoId: fnuNVFgLjHA durationSeconds: 2335 publishedAt: '2026-10-11T15:30:31.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 148 endSeconds: 305 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=148' heldReason: rights - ordinal: 1 text: null startSeconds: 305 endSeconds: 317 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=305' heldReason: rights - ordinal: 2 text: null startSeconds: 317 endSeconds: 375 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=317' heldReason: rights - ordinal: 3 text: null startSeconds: 522 endSeconds: 569 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=522' heldReason: rights - ordinal: 4 text: null startSeconds: 569 endSeconds: 581 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=569' heldReason: rights - ordinal: 5 text: null startSeconds: 581 endSeconds: 657 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=581' heldReason: rights - ordinal: 6 text: null startSeconds: 667 endSeconds: 766 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=667' heldReason: rights - ordinal: 7 text: null startSeconds: 766 endSeconds: 776 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=766' heldReason: rights - ordinal: 8 text: null startSeconds: 1409 endSeconds: 1488 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=1409' heldReason: rights - ordinal: 9 text: null startSeconds: 1499 endSeconds: 1600 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=1499' heldReason: rights - ordinal: 10 text: null startSeconds: 1610 endSeconds: 1726 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=1610' heldReason: rights - ordinal: 11 text: null startSeconds: 1737 endSeconds: 1797 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=1737' heldReason: rights - ordinal: 12 text: null startSeconds: 1808 endSeconds: 1830 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=1808' heldReason: rights - ordinal: 13 text: null startSeconds: 1873 endSeconds: 1885 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=1873' heldReason: rights - ordinal: 14 text: null startSeconds: 1971 endSeconds: 1982 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=1971' heldReason: rights - ordinal: 15 text: null startSeconds: 2004 endSeconds: 2138 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=2004' heldReason: rights - ordinal: 16 text: null startSeconds: 2149 endSeconds: 2277 watch: 'https://liquidgold.cc/watch/fnuNVFgLjHA?t=2149' heldReason: rights --- # Building Self-Learning Loops for Your Agent — Fuad Ali, Arize AI A shopping assistant confidently returns products outside a user's budget because its price filter is broken. Fuad Ali uses the Wonder Toys assistant to show how a production failure can become a complete improvement loop: collect the trace, investigate the code, propose a fix and test it against the cases that failed. The example starts with an application built on the OpenAI Agents SDK and instrumented with Arize. Instead of treating telemetry as a dashboard that a person must inspect, Ali shows how an agent can consume it alongside the application repository and use both to explain what went wrong. Arize Signal groups recurring failures, gathers evidence and opens investigations or pull requests. Arize skills provide a more direct route from a coding agent to traces, datasets and evaluators. Ali then distinguishes a fixed LLM judge rubric from a harness that can inspect tool calls, code and external context while assessing an outcome. The budget example becomes an evaluation that checks whether returned prices actually satisfy the requested range. Agent experiments replay failure datasets against a deployed development endpoint and compare quality, latency and cost before a change ships. Setup pauses and live questions cover keys, tracing and the skills installation. The closing workflow connects automatic detection and repair to validation on real failures, with a reviewable proposal for a human to assess. Speaker info: - https://www.fuadali.com/index.html - https://github.com/Arize-ai/arize-skills - https://github.com/Arize-ai/project-rosetta-stone Timestamps: 0:00 - Self learning loops and telemetry 2:16 - Agents consuming observability 5:02 - A production budget failure 6:09 - The Wonder Toys assistant 7:29 - Repository and tracing setup 8:25 - Investigate the price filter bug 9:19 - Signal and continuous investigation 10:57 - Skills for coding agents 12:37 - Continuous and on demand workflows 13:30 - Workshop setup and keys 19:53 - Inspect the application traces 20:34 - Signal failure clusters and evidence 23:18 - Detect, investigate, and fix 24:45 - Connect the improvement loop 26:34 - Harness as a judge 28:36 - Tool access and external context 29:48 - Evaluate the budget constraint 30:27 - Agent experiments 31:09 - Build and replay failure datasets 32:46 - Compare quality, latency, and cost 33:13 - An autonomous validation loop 35:30 - Instrumentation and skills setup 37:49 - Wrap up --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/fnuNVFgLjHA-youtube-353021d2046547fa53a9 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/building-self-learning-loops-for-your-agent-fuad-al-1f99c2b1e2a1 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.