--- schema: 1 type: video title: >- From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI description: >- An LLM correctness judge rejects all 13 reports from a financial analysis agent. Laurie Voss examines why: the agent used live web research, while the judge graded from its own knowledge without that research context. Supplying the collected sources to a faithfulness evaluator produces a more useful split of six faithful reports and seven unfaithful ones. The notebook builds the agent with the Claude Agent SDK and instruments it with OpenTelemetry and OpenInference to send traces to Arize AX. Reading those traces exposes failed file writes, missing report content, and excessive web searches. A deterministic ticker check adds a cheap first layer of evaluation. Voss builds a custom actionability rubric with explicit criteria, tagged inputs, examples, and binary labels. Each evaluator checks one dimension, so a failure identifies what needs attention. Human annotations provide a comparison set for evaluating the judge, with deliberately random demo labels illustrating how to investigate disagreement. The workshop covers held out tests, precision and recall, judge biases, and the distinction between capability and regression evals. Failing traces become datasets, their explanations guide prompt revisions, and controlled experiments compare the changed agent on the same cases. Online evaluations and production monitoring extend that loop to new traffic, with requirements and recurring failure themes directing subsequent coding agent fixes. The central practice is to read actual traces and define success before automating the grading. Speaker info: - https://x.com/seldo - https://seldo.com - https://github.com/Arize-ai/arize-skills Timestamps: 0:00 - Workshop overview and notebook setup 7:31 - Traces, evals, and the limits of vibes 11:21 - Code, LLM, and human evaluations 14:27 - Agent failures and grading valid solutions 17:18 - Capability and regression evals 21:19 - Instrumenting the Claude Agent SDK 28:47 - Building and tracing the financial analyst 36:26 - Generating diverse test queries 38:54 - Reading traces and defining success 45:31 - Categorizing failures and choosing priorities 49:16 - Writing a deterministic ticker check 58:14 - A correctness judge that rejects every report 62:11 - Faithfulness with the research context 66:25 - Writing a custom actionability rubric 77:35 - Focused evaluators and production criteria 80:00 - Comparing judges with human annotations 86:25 - Precision, recall, and judge biases 90:27 - Turning failures into evaluation datasets 92:39 - Using explanations to improve prompts 95:24 - Comparing changes with controlled experiments 102:29 - Online evaluations and production monitoring 106:01 - Coding agent fixes guided by failure themes 108:27 - Putting the full improvement loop together --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/qZj7sqmidqA-youtube-2e5bce3e8d24a4ed6100 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/from-vibes-to-production-evaluating-and-shipping-ai-ffc926405bec markdown: >- https://liquidgold.cc/channel/aidotengineer/video/from-vibes-to-production-evaluating-and-shipping-ai-ffc926405bec.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:19:51.382Z' language: null children: [] videoId: qZj7sqmidqA durationSeconds: 6687 publishedAt: '2026-10-05T22:00:36.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 458 endSeconds: 469 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=458' heldReason: rights - ordinal: 1 text: null startSeconds: 469 endSeconds: 681 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=469' heldReason: rights - ordinal: 2 text: null startSeconds: 469 endSeconds: 681 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=469' heldReason: rights - ordinal: 3 text: null startSeconds: 691 endSeconds: 874 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=691' heldReason: rights - ordinal: 4 text: null startSeconds: 884 endSeconds: 1047 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=884' heldReason: rights - ordinal: 5 text: null startSeconds: 1047 endSeconds: 1058 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=1047' heldReason: rights - ordinal: 6 text: null startSeconds: 1509 endSeconds: 1595 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=1509' heldReason: rights - ordinal: 7 text: null startSeconds: 1595 endSeconds: 1606 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=1595' heldReason: rights - ordinal: 8 text: null startSeconds: 1924 endSeconds: 2013 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=1924' heldReason: rights - ordinal: 9 text: null startSeconds: 2023 endSeconds: 2161 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=2023' heldReason: rights - ordinal: 10 text: null startSeconds: 2334 endSeconds: 2346 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=2334' heldReason: rights - ordinal: 11 text: null startSeconds: 2437 endSeconds: 2557 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=2437' heldReason: rights - ordinal: 12 text: null startSeconds: 2742 endSeconds: 2959 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=2742' heldReason: rights - ordinal: 13 text: null startSeconds: 2742 endSeconds: 2959 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=2742' heldReason: rights - ordinal: 14 text: null startSeconds: 2959 endSeconds: 2969 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=2959' heldReason: rights - ordinal: 15 text: null startSeconds: 3371 endSeconds: 3501 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=3371' heldReason: rights - ordinal: 16 text: null startSeconds: 3511 endSeconds: 3740 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=3511' heldReason: rights - ordinal: 17 text: null startSeconds: 3750 endSeconds: 3993 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=3750' heldReason: rights - ordinal: 18 text: null startSeconds: 3993 endSeconds: 4004 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=3993' heldReason: rights - ordinal: 19 text: null startSeconds: 4133 endSeconds: 4233 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=4133' heldReason: rights - ordinal: 20 text: null startSeconds: 4245 endSeconds: 4375 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=4245' heldReason: rights - ordinal: 21 text: null startSeconds: 4388 endSeconds: 4492 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=4388' heldReason: rights - ordinal: 22 text: null startSeconds: 4596 endSeconds: 4666 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=4596' heldReason: rights - ordinal: 23 text: null startSeconds: 4676 endSeconds: 4807 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=4676' heldReason: rights - ordinal: 24 text: null startSeconds: 4676 endSeconds: 4807 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=4676' heldReason: rights - ordinal: 25 text: null startSeconds: 4807 endSeconds: 4818 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=4807' heldReason: rights - ordinal: 26 text: null startSeconds: 4964 endSeconds: 5093 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=4964' heldReason: rights - ordinal: 27 text: null startSeconds: 5196 endSeconds: 5207 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=5196' heldReason: rights - ordinal: 28 text: null startSeconds: 5207 endSeconds: 5429 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=5207' heldReason: rights - ordinal: 29 text: null startSeconds: 5441 endSeconds: 5562 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=5441' heldReason: rights - ordinal: 30 text: null startSeconds: 5573 endSeconds: 5733 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=5573' heldReason: rights - ordinal: 31 text: null startSeconds: 5861 endSeconds: 5871 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=5861' heldReason: rights - ordinal: 32 text: null startSeconds: 5973 endSeconds: 6088 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=5973' heldReason: rights - ordinal: 33 text: null startSeconds: 6088 endSeconds: 6099 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=6088' heldReason: rights - ordinal: 34 text: null startSeconds: 6193 endSeconds: 6287 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=6193' heldReason: rights - ordinal: 35 text: null startSeconds: 6298 endSeconds: 6378 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=6298' heldReason: rights - ordinal: 36 text: null startSeconds: 6388 endSeconds: 6469 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=6388' heldReason: rights - ordinal: 37 text: null startSeconds: 6469 endSeconds: 6481 watch: 'https://liquidgold.cc/watch/qZj7sqmidqA?t=6469' heldReason: rights --- # From Vibes to Production: Evaluating and Shipping AI Agents That Work 101 — Laurie Voss, Arize AI An LLM correctness judge rejects all 13 reports from a financial analysis agent. Laurie Voss examines why: the agent used live web research, while the judge graded from its own knowledge without that research context. Supplying the collected sources to a faithfulness evaluator produces a more useful split of six faithful reports and seven unfaithful ones. The notebook builds the agent with the Claude Agent SDK and instruments it with OpenTelemetry and OpenInference to send traces to Arize AX. Reading those traces exposes failed file writes, missing report content, and excessive web searches. A deterministic ticker check adds a cheap first layer of evaluation. Voss builds a custom actionability rubric with explicit criteria, tagged inputs, examples, and binary labels. Each evaluator checks one dimension, so a failure identifies what needs attention. Human annotations provide a comparison set for evaluating the judge, with deliberately random demo labels illustrating how to investigate disagreement. The workshop covers held out tests, precision and recall, judge biases, and the distinction between capability and regression evals. Failing traces become datasets, their explanations guide prompt revisions, and controlled experiments compare the changed agent on the same cases. Online evaluations and production monitoring extend that loop to new traffic, with requirements and recurring failure themes directing subsequent coding agent fixes. The central practice is to read actual traces and define success before automating the grading. Speaker info: - https://x.com/seldo - https://seldo.com - https://github.com/Arize-ai/arize-skills Timestamps: 0:00 - Workshop overview and notebook setup 7:31 - Traces, evals, and the limits of vibes 11:21 - Code, LLM, and human evaluations 14:27 - Agent failures and grading valid solutions 17:18 - Capability and regression evals 21:19 - Instrumenting the Claude Agent SDK 28:47 - Building and tracing the financial analyst 36:26 - Generating diverse test queries 38:54 - Reading traces and defining success 45:31 - Categorizing failures and choosing priorities 49:16 - Writing a deterministic ticker check 58:14 - A correctness judge that rejects every report 62:11 - Faithfulness with the research context 66:25 - Writing a custom actionability rubric 77:35 - Focused evaluators and production criteria 80:00 - Comparing judges with human annotations 86:25 - Precision, recall, and judge biases 90:27 - Turning failures into evaluation datasets 92:39 - Using explanations to improve prompts 95:24 - Comparing changes with controlled experiments 102:29 - Online evaluations and production monitoring 106:01 - Coding agent fixes guided by failure themes 108:27 - Putting the full improvement loop together --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/qZj7sqmidqA-youtube-2e5bce3e8d24a4ed6100 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/from-vibes-to-production-evaluating-and-shipping-ai-ffc926405bec · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.