--- schema: 1 type: video title: >- How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI description: >- Your AI agent wrote a confident stock report saying 116% of analysts agree. Nothing in your checks caught it. This is the workshop for fixing that. Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm) builds a full evals loop on a Claude-powered financial analyst agent: trace it with Arize Phoenix, read the traces, write code evals and LLM-as-a-judge evals, test whether the judge can be trusted, prove a prompt fix with experiments, upgrade from Claude Haiku 4.5 to 5.5 with evidence, and gate it all in CI with npm test. Speaker: Laurie Voss, Head of Developer Relations, Arize AI X: https://x.com/seldo https://arize.com Phoenix: https://phoenix.arize.com 0:00 Intro & setup 2:47 "Looks good to me" isn't a test suite 4:57 Traces vs. evals: the mental model 8:30 Code evals, LLM judges & human review 11:57 Why agents are confidently wrong 16:32 Step 1: Tracing with Phoenix 36:36 Step 2: Reading your traces 43:21 The 116% analyst consensus bug 50:36 Step 3: Code evals 56:11 Step 4: LLM-as-a-judge 1:00:30 Why the correctness eval failed every report 1:10:41 Step 5: Writing your own judge 1:24:45 Step 6: Testing your judge 1:38:55 Step 7: Experiments 1:48:30 Upgrading to Claude Haiku 5.5 1:50:55 Evals in CI 1:55:25 Where to start on Monday Live from the Sheraton New York Times Square Hotel in New York City, AI Engineer New York 2026 is an AI conference for engineers and technical leaders building production AI across banking, hedge funds, fintech, insurance, and asset management. Event: AI Engineer NYC Dates: October 12–14, 2026 Venue: Sheraton New York Times Square Hotel Learn more about the event: https://ai.engineer/nyc/2026# Subscribe for more AI Engineer talks, livestreams, and event coverage: https://www.youtube.com/@aiDotEngineer canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/how-to-know-if-your-ai-agent-is-any-good-evals-with-efa3e05f5f83 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/how-to-know-if-your-ai-agent-is-any-good-evals-with-efa3e05f5f83.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:23:32.244Z' language: null children: [] videoId: SHMCGrNJJGc durationSeconds: 6973 publishedAt: '2026-10-17T19:00:23.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 183 endSeconds: 304 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=183' heldReason: rights - ordinal: 1 text: null startSeconds: 315 endSeconds: 515 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=315' heldReason: rights - ordinal: 2 text: null startSeconds: 315 endSeconds: 515 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=315' heldReason: rights - ordinal: 3 text: null startSeconds: 526 endSeconds: 718 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=526' heldReason: rights - ordinal: 4 text: null startSeconds: 526 endSeconds: 718 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=526' heldReason: rights - ordinal: 5 text: null startSeconds: 718 endSeconds: 729 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=718' heldReason: rights - ordinal: 6 text: null startSeconds: 729 endSeconds: 1002 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=729' heldReason: rights - ordinal: 7 text: null startSeconds: 729 endSeconds: 1002 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=729' heldReason: rights - ordinal: 8 text: null startSeconds: 1002 endSeconds: 1014 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=1002' heldReason: rights - ordinal: 9 text: null startSeconds: 1134 endSeconds: 1256 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=1134' heldReason: rights - ordinal: 10 text: null startSeconds: 1267 endSeconds: 1351 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=1267' heldReason: rights - ordinal: 11 text: null startSeconds: 1460 endSeconds: 1472 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=1460' heldReason: rights - ordinal: 12 text: null startSeconds: 1613 endSeconds: 1721 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=1613' heldReason: rights - ordinal: 13 text: null startSeconds: 1733 endSeconds: 1839 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=1733' heldReason: rights - ordinal: 14 text: null startSeconds: 1850 endSeconds: 1953 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=1850' heldReason: rights - ordinal: 15 text: null startSeconds: 1963 endSeconds: 2067 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=1963' heldReason: rights - ordinal: 16 text: null startSeconds: 2078 endSeconds: 2200 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=2078' heldReason: rights - ordinal: 17 text: null startSeconds: 2213 endSeconds: 2295 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=2213' heldReason: rights - ordinal: 18 text: null startSeconds: 2307 endSeconds: 2430 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=2307' heldReason: rights - ordinal: 19 text: null startSeconds: 2620 endSeconds: 2733 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=2620' heldReason: rights - ordinal: 20 text: null startSeconds: 2743 endSeconds: 2882 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=2743' heldReason: rights - ordinal: 21 text: null startSeconds: 2743 endSeconds: 2882 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=2743' heldReason: rights - ordinal: 22 text: null startSeconds: 3012 endSeconds: 3023 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=3012' heldReason: rights - ordinal: 23 text: null startSeconds: 3060 endSeconds: 3149 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=3060' heldReason: rights - ordinal: 24 text: null startSeconds: 3287 endSeconds: 3376 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=3287' heldReason: rights - ordinal: 25 text: null startSeconds: 3388 endSeconds: 3638 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=3388' heldReason: rights - ordinal: 26 text: null startSeconds: 3648 endSeconds: 3733 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=3648' heldReason: rights - ordinal: 27 text: null startSeconds: 3733 endSeconds: 3743 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=3733' heldReason: rights - ordinal: 28 text: null startSeconds: 3964 endSeconds: 3976 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=3964' heldReason: rights - ordinal: 29 text: null startSeconds: 4187 endSeconds: 4244 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=4187' heldReason: rights - ordinal: 30 text: null startSeconds: 4255 endSeconds: 4383 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=4255' heldReason: rights - ordinal: 31 text: null startSeconds: 4393 endSeconds: 4528 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=4393' heldReason: rights - ordinal: 32 text: null startSeconds: 4538 endSeconds: 4648 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=4538' heldReason: rights - ordinal: 33 text: null startSeconds: 4800 endSeconds: 4926 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=4800' heldReason: rights - ordinal: 34 text: null startSeconds: 4936 endSeconds: 5068 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=4936' heldReason: rights - ordinal: 35 text: null startSeconds: 5068 endSeconds: 5078 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=5068' heldReason: rights - ordinal: 36 text: null startSeconds: 5100 endSeconds: 5190 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=5100' heldReason: rights - ordinal: 37 text: null startSeconds: 5202 endSeconds: 5334 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=5202' heldReason: rights - ordinal: 38 text: null startSeconds: 5202 endSeconds: 5334 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=5202' heldReason: rights - ordinal: 39 text: null startSeconds: 5345 endSeconds: 5430 watch: 'https://liquidgold.cc/watch/SHMCGrNJJGc?t=5345' heldReason: rights --- # How to Know If Your AI Agent Is Any Good: Evals With Phoenix & Claude — Laurie Voss, Arize AI Your AI agent wrote a confident stock report saying 116% of analysts agree. Nothing in your checks caught it. This is the workshop for fixing that. Laurie Voss (Head of Developer Relations at Arize AI, co-founder of npm) builds a full evals loop on a Claude-powered financial analyst agent: trace it with Arize Phoenix, read the traces, write code evals and LLM-as-a-judge evals, test whether the judge can be trusted, prove a prompt fix with experiments, upgrade from Claude Haiku 4.5 to 5.5 with evidence, and gate it all in CI with npm test. Speaker: Laurie Voss, Head of Developer Relations, Arize AI X: https://x.com/seldo https://arize.com Phoenix: https://phoenix.arize.com 0:00 Intro & setup 2:47 "Looks good to me" isn't a test suite 4:57 Traces vs. evals: the mental model 8:30 Code evals, LLM judges & human review 11:57 Why agents are confidently wrong 16:32 Step 1: Tracing with Phoenix 36:36 Step 2: Reading your traces 43:21 The 116% analyst consensus bug 50:36 Step 3: Code evals 56:11 Step 4: LLM-as-a-judge 1:00:30 Why the correctness eval failed every report 1:10:41 Step 5: Writing your own judge 1:24:45 Step 6: Testing your judge 1:38:55 Step 7: Experiments 1:48:30 Upgrading to Claude Haiku 5.5 1:50:55 Evals in CI 1:55:25 Where to start on Monday Live from the Sheraton New York Times Square Hotel in New York City, AI Engineer New York 2026 is an AI conference for engineers and technical leaders building production AI across banking, hedge funds, fintech, insurance, and asset management. Event: AI Engineer NYC Dates: October 12–14, 2026 Venue: Sheraton New York Times Square Hotel Learn more about the event: https://ai.engineer/nyc/2026# Subscribe for more AI Engineer talks, livestreams, and event coverage: https://www.youtube.com/@aiDotEngineer Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/how-to-know-if-your-ai-agent-is-any-good-evals-with-efa3e05f5f83 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:12 to 2:52) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (2:52 to 5:04) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (5:04 to 8:35) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (8:35 to 11:58) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (11:58 to 16:42) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (16:42 to 18:44) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (18:44 to 20:56) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (20:56 to 22:31) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 8: Document excerpt (22:31 to 24:20) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 9: Document excerpt (24:20 to 26:43) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 10: Document excerpt (26:43 to 28:41) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 11: Document excerpt (28:41 to 30:39) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 12: Document excerpt (30:39 to 32:33) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 13: Document excerpt (32:33 to 34:27) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 14: Document excerpt (34:27 to 36:40) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 15: Document excerpt (36:40 to 38:15) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 16: Document excerpt (38:15 to 40:30) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 17: Document excerpt (40:30 to 42:44) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 18: Document excerpt (42:44 to 43:29) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 19: Document excerpt (43:29 to 45:33) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 20: Document excerpt (45:33 to 48:02) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 21: Document excerpt (48:02 to 50:12) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 22: Document excerpt (50:12 to 50:48) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 23: Document excerpt (50:48 to 52:29) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 24: Document excerpt (52:29 to 54:35) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 25: Document excerpt (54:35 to 56:16) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 26: Document excerpt (56:16 to 60:38) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 27: Document excerpt (60:38 to 62:13) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 28: Document excerpt (62:13 to 63:58) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 29: Document excerpt (63:58 to 66:04) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 30: Document excerpt (66:04 to 67:38) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 31: Document excerpt (67:38 to 69:34) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 32: Document excerpt (69:34 to 70:44) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 33: Document excerpt (70:44 to 73:03) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 34: Document excerpt (73:03 to 75:28) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 35: Document excerpt (75:28 to 77:28) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 36: Document excerpt (77:28 to 79:49) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 37: Document excerpt (79:49 to 82:06) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 38: Document excerpt (82:06 to 84:28) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 39: Document excerpt (84:28 to 84:49) > Held: transcript rights do not permit this chunk's text.