--- schema: 1 type: video title: >- From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI description: >- A toy store agent fails a request for toys under $6 because its search tool has no price filter. Laurie Voss turns that missing feature into a live demonstration of an improvement loop: add observability, inspect what happened, find the problem, and ask a coding agent to fix it. The Wonder Toys application starts without traces flowing into Arize AX. Skills installed in the workshop repository help the coding agent add tracing and query the resulting data. The diagnosis reaches beyond exceptions and HTTP errors to search quality, including empty results and overly narrow filters. After adding price filtering, Voss repeats the shopping request and checks the new input and output in the trace view. The broader argument is that reading traces individually stops working when an application produces millions of them. Evals compress behavior into scores and explanations, but people can still become the bottleneck when they must interpret every failure. Voss introduces Signal as another layer that groups recurring problems and suggests fixes. In the demonstration, findings can become GitHub issues, evaluation datasets, new evaluators, or proposed pull requests. Audience questions press on deployment, trust, and preventing a fix from breaking behavior that already works. Regression evals provide that check, while Signal itself has tracing and an evaluation suite. The workshop moves from a manually requested repair toward continuous improvement, with definitions of good behavior directing what the automation should change. Speaker info: - https://x.com/seldo - https://seldo.com - https://github.com/Arize-ai/project-rosetta-stone Timestamps: 0:00 - From 101 to continuous improvement 3:12 - Traces as the source of truth 6:13 - Scaling beyond manual trace review 9:02 - Finding signals and closing the loop 12:03 - Workshop setup 14:13 - Wonder Toys and trust questions 16:31 - Adding observability with a coding agent 20:19 - Testing and inspecting live traces 22:25 - Exposing the missing price filter 24:41 - Asking an agent to analyze traces 28:51 - Diagnosing search quality problems 30:28 - Adding price filtering 33:05 - Continuous monitoring with Signal 36:06 - Deployment and regression evaluation questions 40:22 - Evaluating Signal itself --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/F0TNSmbo5hE-youtube-69b6c7350f3746e53f73 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/from-vibes-to-production-evaluating-and-shipping-ai-d7c6e7232474 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/from-vibes-to-production-evaluating-and-shipping-ai-d7c6e7232474.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:19:50.532Z' language: null children: [] videoId: F0TNSmbo5hE durationSeconds: 2537 publishedAt: '2026-10-05T23:00:09.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 210 endSeconds: 375 watch: 'https://liquidgold.cc/watch/F0TNSmbo5hE?t=210' heldReason: rights - ordinal: 1 text: null startSeconds: 375 endSeconds: 387 watch: 'https://liquidgold.cc/watch/F0TNSmbo5hE?t=375' heldReason: rights - ordinal: 2 text: null startSeconds: 558 endSeconds: 731 watch: 'https://liquidgold.cc/watch/F0TNSmbo5hE?t=558' heldReason: rights - ordinal: 3 text: null startSeconds: 558 endSeconds: 731 watch: 'https://liquidgold.cc/watch/F0TNSmbo5hE?t=558' heldReason: rights - ordinal: 4 text: null startSeconds: 1012 endSeconds: 1219 watch: 'https://liquidgold.cc/watch/F0TNSmbo5hE?t=1012' heldReason: rights - ordinal: 5 text: null startSeconds: 1357 endSeconds: 1481 watch: 'https://liquidgold.cc/watch/F0TNSmbo5hE?t=1357' heldReason: rights - ordinal: 6 text: null startSeconds: 1739 endSeconds: 1750 watch: 'https://liquidgold.cc/watch/F0TNSmbo5hE?t=1739' heldReason: rights - ordinal: 7 text: null startSeconds: 1853 endSeconds: 1990 watch: 'https://liquidgold.cc/watch/F0TNSmbo5hE?t=1853' heldReason: rights - ordinal: 8 text: null startSeconds: 2000 endSeconds: 2174 watch: 'https://liquidgold.cc/watch/F0TNSmbo5hE?t=2000' heldReason: rights - ordinal: 9 text: null startSeconds: 2186 endSeconds: 2429 watch: 'https://liquidgold.cc/watch/F0TNSmbo5hE?t=2186' heldReason: rights - ordinal: 10 text: null startSeconds: 2441 endSeconds: 2537 watch: 'https://liquidgold.cc/watch/F0TNSmbo5hE?t=2441' heldReason: rights --- # From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI A toy store agent fails a request for toys under $6 because its search tool has no price filter. Laurie Voss turns that missing feature into a live demonstration of an improvement loop: add observability, inspect what happened, find the problem, and ask a coding agent to fix it. The Wonder Toys application starts without traces flowing into Arize AX. Skills installed in the workshop repository help the coding agent add tracing and query the resulting data. The diagnosis reaches beyond exceptions and HTTP errors to search quality, including empty results and overly narrow filters. After adding price filtering, Voss repeats the shopping request and checks the new input and output in the trace view. The broader argument is that reading traces individually stops working when an application produces millions of them. Evals compress behavior into scores and explanations, but people can still become the bottleneck when they must interpret every failure. Voss introduces Signal as another layer that groups recurring problems and suggests fixes. In the demonstration, findings can become GitHub issues, evaluation datasets, new evaluators, or proposed pull requests. Audience questions press on deployment, trust, and preventing a fix from breaking behavior that already works. Regression evals provide that check, while Signal itself has tracing and an evaluation suite. The workshop moves from a manually requested repair toward continuous improvement, with definitions of good behavior directing what the automation should change. Speaker info: - https://x.com/seldo - https://seldo.com - https://github.com/Arize-ai/project-rosetta-stone Timestamps: 0:00 - From 101 to continuous improvement 3:12 - Traces as the source of truth 6:13 - Scaling beyond manual trace review 9:02 - Finding signals and closing the loop 12:03 - Workshop setup 14:13 - Wonder Toys and trust questions 16:31 - Adding observability with a coding agent 20:19 - Testing and inspecting live traces 22:25 - Exposing the missing price filter 24:41 - Asking an agent to analyze traces 28:51 - Diagnosing search quality problems 30:28 - Adding price filtering 33:05 - Continuous monitoring with Signal 36:06 - Deployment and regression evaluation questions 40:22 - Evaluating Signal itself --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/F0TNSmbo5hE-youtube-69b6c7350f3746e53f73 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/from-vibes-to-production-evaluating-and-shipping-ai-d7c6e7232474 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.