--- schema: 1 type: video title: 'The maturity phases of running evals — Phil Hetzel, Braintrust' description: >- Most teams approach evals like unit tests and try to cover every possible failure. Phil Hetzel from Braintrust argues that is the wrong frame: enumerate your known failure modes, cover those specifically, and ship. The goal is a flywheel where production traces surface what is going wrong, feed back into offline experimentation, and guide the next improvement. The session walks four maturity stages: vibe checking with documented human justifications not just thumbs up or down, LLM as judge built from those justifications at scale, then the hard part, tool calls that touch external systems. Context gathering tools are manageable. CRUD tools are not, because you have to represent the state of external systems at the exact moment the original trace ran. Timestamp queries against a vector database and injecting captured system state directly into the trace are two approaches for getting there. Speaker info: - https://www.linkedin.com/in/philliphetzel/ --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/FB-MLPhL9Ms-the-maturity-phases-of-running-evals Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/the-maturity-phases-of-running-evals-phil-hetzel-br-1e900ca10680 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/the-maturity-phases-of-running-evals-phil-hetzel-br-1e900ca10680.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T13:06:08.811Z' language: null children: [] videoId: FB-MLPhL9Ms durationSeconds: 1114 publishedAt: '2026-05-27T13:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 25 endSeconds: 164 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=25' heldReason: rights - ordinal: 1 text: null startSeconds: 164 endSeconds: 175 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=164' heldReason: rights - ordinal: 2 text: null startSeconds: 175 endSeconds: 300 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=175' heldReason: rights - ordinal: 3 text: null startSeconds: 175 endSeconds: 300 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=175' heldReason: rights - ordinal: 4 text: null startSeconds: 300 endSeconds: 310 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=300' heldReason: rights - ordinal: 5 text: null startSeconds: 310 endSeconds: 391 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=310' heldReason: rights - ordinal: 6 text: null startSeconds: 403 endSeconds: 503 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=403' heldReason: rights - ordinal: 7 text: null startSeconds: 403 endSeconds: 503 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=403' heldReason: rights - ordinal: 8 text: null startSeconds: 515 endSeconds: 596 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=515' heldReason: rights - ordinal: 9 text: null startSeconds: 596 endSeconds: 608 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=596' heldReason: rights - ordinal: 10 text: null startSeconds: 608 endSeconds: 725 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=608' heldReason: rights - ordinal: 11 text: null startSeconds: 608 endSeconds: 725 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=608' heldReason: rights - ordinal: 12 text: null startSeconds: 608 endSeconds: 725 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=608' heldReason: rights - ordinal: 13 text: null startSeconds: 608 endSeconds: 725 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=608' heldReason: rights - ordinal: 14 text: null startSeconds: 735 endSeconds: 834 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=735' heldReason: rights - ordinal: 15 text: null startSeconds: 735 endSeconds: 834 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=735' heldReason: rights - ordinal: 16 text: null startSeconds: 735 endSeconds: 834 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=735' heldReason: rights - ordinal: 17 text: null startSeconds: 834 endSeconds: 846 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=834' heldReason: rights - ordinal: 18 text: null startSeconds: 846 endSeconds: 944 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=846' heldReason: rights - ordinal: 19 text: null startSeconds: 846 endSeconds: 944 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=846' heldReason: rights - ordinal: 20 text: null startSeconds: 955 endSeconds: 1114 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=955' heldReason: rights - ordinal: 21 text: null startSeconds: 955 endSeconds: 1114 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=955' heldReason: rights - ordinal: 22 text: null startSeconds: 955 endSeconds: 1114 watch: 'https://liquidgold.cc/watch/FB-MLPhL9Ms?t=955' heldReason: rights --- # The maturity phases of running evals — Phil Hetzel, Braintrust Most teams approach evals like unit tests and try to cover every possible failure. Phil Hetzel from Braintrust argues that is the wrong frame: enumerate your known failure modes, cover those specifically, and ship. The goal is a flywheel where production traces surface what is going wrong, feed back into offline experimentation, and guide the next improvement. The session walks four maturity stages: vibe checking with documented human justifications not just thumbs up or down, LLM as judge built from those justifications at scale, then the hard part, tool calls that touch external systems. Context gathering tools are manageable. CRUD tools are not, because you have to represent the state of external systems at the exact moment the original trace ran. Timestamp queries against a vector database and injecting captured system state directly into the trace are two approaches for getting there. Speaker info: - https://www.linkedin.com/in/philliphetzel/ --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/FB-MLPhL9Ms-the-maturity-phases-of-running-evals Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/the-maturity-phases-of-running-evals-phil-hetzel-br-1e900ca10680 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.