---
schema: 1
type: video
title: 'Your Agent Evolved. Your Evals Didn''t. — Ameya Bhatawdekar, Braintrust'
description: >-
  Teams built orchestration graphs because the models of 2024 could not be
  trusted to orchestrate, and then the models learned to orchestrate and the
  graphs became the thing holding them back. Ameya Bhatawdekar traces that loop
  across five generations of architecture, each one forced by a step change in
  model capability, and argues that evals have to move with it. A single prompt
  needed only answer quality. A retrieval chain added a parser that grabs the
  wrong field and a retriever that returns the wrong context. Graphs added
  branch logic, contracts between nodes, and classifier nodes that misfire
  quietly, which is a great deal of new surface to check. What changed most
  recently is not another layer but the unit of measurement. Once a loop is
  reliable enough to run free, the same input produces visibly different
  trajectories on every run, so a single eval result stops meaning very much. He
  separates the two questions it hides. Pass at k asks whether the system
  succeeds at least once across k attempts, which measures capability. The
  stricter variant asks how many of those k attempts succeed, which measures
  reliability. A system can look strong on the first and weak on the second, and
  you only learn that by running the distribution rather than the sample.
  Underneath it all is the claim that evals are the durable asset across
  replatformings, and that the ones which go stale are the ones nobody feeds
  from production. Speaker info: - https://www.linkedin.com/in/ameyab
  Timestamps: 0:00 - Replatforming, not iterating 2:24 - Step function model
  changes, not incremental ones 3:30 - Why you cannot just drop in a new model
  5:34 - Grounding it in a site reliability agent 6:40 - One prompt, one call,
  one answer to grade 7:44 - The chain, and where retrieval goes wrong 8:48 -
  The first ReAct loop, and why it fell short 9:49 - Taking control back with
  workflow graphs 11:58 - The eval surface a graph creates 13:07 - Models get
  good enough to loop again 15:19 - Pass at k for capability, the stricter one
  for reliability 16:21 - Memory, sandboxes, and skills around the loop 19:34 -
  The flywheel most teams accept but never run
canonical: >-
  https://liquidgold.cc/channel/aidotengineer/explore/video/your-agent-evolved-your-evals-didn-t-ameya-bhatawde-0f30a69a0632
markdown: >-
  https://liquidgold.cc/channel/aidotengineer/video/your-agent-evolved-your-evals-didn-t-ameya-bhatawde-0f30a69a0632.md
llms: null
parent: 'https://liquidgold.cc/channel/aidotengineer.md'
channel: aidotengineer
updated: '2026-10-24T05:02:00.229Z'
language: null
children: []
videoId: nxokqOq1imY
durationSeconds: 1453
publishedAt: '2026-08-20T13:00:06.000Z'
searchChunks:
  - ordinal: 0
    text: null
    startSeconds: 12
    endSeconds: 22
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=12'
    heldReason: rights
  - ordinal: 1
    text: null
    startSeconds: 22
    endSeconds: 155
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=22'
    heldReason: rights
  - ordinal: 2
    text: null
    startSeconds: 166
    endSeconds: 210
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=166'
    heldReason: rights
  - ordinal: 3
    text: null
    startSeconds: 222
    endSeconds: 342
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=222'
    heldReason: rights
  - ordinal: 4
    text: null
    startSeconds: 342
    endSeconds: 352
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=342'
    heldReason: rights
  - ordinal: 5
    text: null
    startSeconds: 352
    endSeconds: 400
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=352'
    heldReason: rights
  - ordinal: 6
    text: null
    startSeconds: 412
    endSeconds: 469
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=412'
    heldReason: rights
  - ordinal: 7
    text: null
    startSeconds: 485
    endSeconds: 533
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=485'
    heldReason: rights
  - ordinal: 8
    text: null
    startSeconds: 543
    endSeconds: 589
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=543'
    heldReason: rights
  - ordinal: 9
    text: null
    startSeconds: 589
    endSeconds: 599
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=589'
    heldReason: rights
  - ordinal: 10
    text: null
    startSeconds: 722
    endSeconds: 733
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=722'
    heldReason: rights
  - ordinal: 11
    text: null
    startSeconds: 804
    endSeconds: 919
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=804'
    heldReason: rights
  - ordinal: 12
    text: null
    startSeconds: 919
    endSeconds: 931
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=919'
    heldReason: rights
  - ordinal: 13
    text: null
    startSeconds: 931
    endSeconds: 990
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=931'
    heldReason: rights
  - ordinal: 14
    text: null
    startSeconds: 1001
    endSeconds: 1174
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=1001'
    heldReason: rights
  - ordinal: 15
    text: null
    startSeconds: 1187
    endSeconds: 1453
    watch: 'https://liquidgold.cc/watch/nxokqOq1imY?t=1187'
    heldReason: rights
---


# Your Agent Evolved. Your Evals Didn't. — Ameya Bhatawdekar, Braintrust

Teams built orchestration graphs because the models of 2024 could not be trusted to orchestrate, and then the models learned to orchestrate and the graphs became the thing holding them back. Ameya Bhatawdekar traces that loop across five generations of architecture, each one forced by a step change in model capability, and argues that evals have to move with it. A single prompt needed only answer quality. A retrieval chain added a parser that grabs the wrong field and a retriever that returns the wrong context. Graphs added branch logic, contracts between nodes, and classifier nodes that misfire quietly, which is a great deal of new surface to check. What changed most recently is not another layer but the unit of measurement. Once a loop is reliable enough to run free, the same input produces visibly different trajectories on every run, so a single eval result stops meaning very much. He separates the two questions it hides. Pass at k asks whether the system succeeds at least once across k attempts, which measures capability. The stricter variant asks how many of those k attempts succeed, which measures reliability. A system can look strong on the first and weak on the second, and you only learn that by running the distribution rather than the sample. Underneath it all is the claim that evals are the durable asset across replatformings, and that the ones which go stale are the ones nobody feeds from production. Speaker info: - https://www.linkedin.com/in/ameyab Timestamps: 0:00 - Replatforming, not iterating 2:24 - Step function model changes, not incremental ones 3:30 - Why you cannot just drop in a new model 5:34 - Grounding it in a site reliability agent 6:40 - One prompt, one call, one answer to grade 7:44 - The chain, and where retrieval goes wrong 8:48 - The first ReAct loop, and why it fell short 9:49 - Taking control back with workflow graphs 11:58 - The eval surface a graph creates 13:07 - Models get good enough to loop again 15:19 - Pass at k for capability, the stricter one for reliability 16:21 - Memory, sandboxes, and skills around the loop 19:34 - The flywheel most teams accept but never run

Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/your-agent-evolved-your-evals-didn-t-ameya-bhatawde-0f30a69a0632 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md)

> Search chunk text is held until public-source provenance is verified.

## Document chunks

> Missing: no stored document chunks are available.