--- schema: 1 type: video title: Why Building an Eval Platform Is Harder Than It Looks — Braintrust description: >- Most teams start their evals in a spreadsheet. Here's what happens when they try to grow out of it. Hossein Niazmandi, who leads solutions engineering in the West at Braintrust, explains why measuring agent quality is much more than putting a UI on a spreadsheet. He covers the two pillars of agent quality, evals before production and observability after, and why non-deterministic LLMs make both necessary. Then he walks through the stages teams go through: spreadsheets, a custom UI, side-by-side experiments for PMs and domain experts, and finally a flywheel that turns production failures into test cases. At that point the hard part becomes the data: traces that are huge, nested JSON, needing both real-time and long-running queries, which is why Braintrust built BTQL. He closes with agents running evals themselves and surfacing failures nobody thought to look for. In this talk: • Evals vs observability, and why you need both • The stages of eval maturity, from spreadsheets to a production flywheel • Why agent traces break typical data warehouses • Letting coding agents run evals while humans review the results SPEAKER Hossein Niazmandi, Solutions Engineering, Braintrust LinkedIn: https://www.linkedin.com/in/hossein-niazmandi-20a184bb/ LINKS Braintrust: https://www.braintrust.dev Braintrust SQL / BTQL reference: https://www.braintrust.dev/docs/reference/sql Braintrust on X: https://x.com/braintrust CHAPTERS 0:00 Intro 0:52 What Braintrust does 1:12 Evals and observability 2:21 Why evals matter 3:51 Isn't it just a spreadsheet? 5:11 What's under the iceberg 6:55 The improvement loop 7:35 Stage 1: spreadsheets 9:00 Stage 2: a custom UI 9:50 Stage 3: experiments for everyone 10:20 Stage 4: connecting production 11:49 Agent traces are huge and messy 13:19 Querying with BTQL 14:39 Agents running evals 15:53 Finding the unknown unknowns 16:43 Agents iterate, humans decide Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMEvals #AIAgents #AIEngineer canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/why-building-an-eval-platform-is-harder-than-it-loo-a3025bb251b4 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/why-building-an-eval-platform-is-harder-than-it-loo-a3025bb251b4.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:19:46.100Z' language: null children: [] videoId: mUQoVz7THu0 durationSeconds: 1067 publishedAt: '2026-10-06T18:30:35.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 12 endSeconds: 57 watch: 'https://liquidgold.cc/watch/mUQoVz7THu0?t=12' heldReason: rights - ordinal: 1 text: null startSeconds: 57 endSeconds: 69 watch: 'https://liquidgold.cc/watch/mUQoVz7THu0?t=57' heldReason: rights - ordinal: 2 text: null startSeconds: 69 endSeconds: 82 watch: 'https://liquidgold.cc/watch/mUQoVz7THu0?t=69' heldReason: rights - ordinal: 3 text: null startSeconds: 82 endSeconds: 94 watch: 'https://liquidgold.cc/watch/mUQoVz7THu0?t=82' heldReason: rights - ordinal: 4 text: null startSeconds: 94 endSeconds: 150 watch: 'https://liquidgold.cc/watch/mUQoVz7THu0?t=94' heldReason: rights - ordinal: 5 text: null startSeconds: 150 endSeconds: 161 watch: 'https://liquidgold.cc/watch/mUQoVz7THu0?t=150' heldReason: rights - ordinal: 6 text: null startSeconds: 161 endSeconds: 233 watch: 'https://liquidgold.cc/watch/mUQoVz7THu0?t=161' heldReason: rights - ordinal: 7 text: null startSeconds: 233 endSeconds: 243 watch: 'https://liquidgold.cc/watch/mUQoVz7THu0?t=233' heldReason: rights - ordinal: 8 text: null startSeconds: 243 endSeconds: 321 watch: 'https://liquidgold.cc/watch/mUQoVz7THu0?t=243' heldReason: rights - ordinal: 9 text: null startSeconds: 321 endSeconds: 331 watch: 'https://liquidgold.cc/watch/mUQoVz7THu0?t=321' heldReason: rights - ordinal: 10 text: null startSeconds: 331 endSeconds: 421 watch: 'https://liquidgold.cc/watch/mUQoVz7THu0?t=331' heldReason: rights --- # Why Building an Eval Platform Is Harder Than It Looks — Braintrust Most teams start their evals in a spreadsheet. Here's what happens when they try to grow out of it. Hossein Niazmandi, who leads solutions engineering in the West at Braintrust, explains why measuring agent quality is much more than putting a UI on a spreadsheet. He covers the two pillars of agent quality, evals before production and observability after, and why non-deterministic LLMs make both necessary. Then he walks through the stages teams go through: spreadsheets, a custom UI, side-by-side experiments for PMs and domain experts, and finally a flywheel that turns production failures into test cases. At that point the hard part becomes the data: traces that are huge, nested JSON, needing both real-time and long-running queries, which is why Braintrust built BTQL. He closes with agents running evals themselves and surfacing failures nobody thought to look for. In this talk: • Evals vs observability, and why you need both • The stages of eval maturity, from spreadsheets to a production flywheel • Why agent traces break typical data warehouses • Letting coding agents run evals while humans review the results SPEAKER Hossein Niazmandi, Solutions Engineering, Braintrust LinkedIn: https://www.linkedin.com/in/hossein-niazmandi-20a184bb/ LINKS Braintrust: https://www.braintrust.dev Braintrust SQL / BTQL reference: https://www.braintrust.dev/docs/reference/sql Braintrust on X: https://x.com/braintrust CHAPTERS 0:00 Intro 0:52 What Braintrust does 1:12 Evals and observability 2:21 Why evals matter 3:51 Isn't it just a spreadsheet? 5:11 What's under the iceberg 6:55 The improvement loop 7:35 Stage 1: spreadsheets 9:00 Stage 2: a custom UI 9:50 Stage 3: experiments for everyone 10:20 Stage 4: connecting production 11:49 Agent traces are huge and messy 13:19 Querying with BTQL 14:39 Agents running evals 15:53 Finding the unknown unknowns 16:43 Agents iterate, humans decide Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMEvals #AIAgents #AIEngineer Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/why-building-an-eval-platform-is-harder-than-it-loo-a3025bb251b4 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:12 to 1:09) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (1:09 to 2:30) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (2:30 to 3:53) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (3:53 to 5:21) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (5:21 to 7:01) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (7:01 to 7:37) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (7:37 to 9:07) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (9:07 to 9:55) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 8: Document excerpt (9:55 to 10:29) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 9: Document excerpt (10:29 to 11:58) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 10: Document excerpt (11:58 to 13:28) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 11: Document excerpt (13:28 to 14:46) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 12: Document excerpt (14:46 to 15:54) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 13: Document excerpt (15:54 to 16:43) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 14: Document excerpt (16:43 to 17:47) > Held: transcript rights do not permit this chunk's text.