Talk by Aparna Dhinakaran (Arize AI) arguing that evaluation methods must evolve alongside the agents they grade. It traces the progression from deterministic checks that catch predefined failures, to LLM-as-a-judge for analysis beyond fixed rules, to agent-as-a-judge, which can discover unanticipated failure modes and even open pull requests to fix issues found. Structured timestamps cover why evals became harder as agents gained reasoning, tool calls, and multi-step loops, and where evals go next. # Video Presentation Observations — Aparna Dhinakaran, Arize AI ## Video Context - **Format**: Conference/track talk introduction — speaker introduces herself as "one of the founders of Arize" and frames an "evals track" lineup, suggesting this is a brief keynote-style session rather than a full tutorial. - **Topic**: "The future of evals," pivoting from **LLM-as-a-judge** to **agent-as-a-judge** evaluation approaches. ## Explicit Self-Description Claims - **Company role**: > "My name is Aparna, one of the founders of Arize. We work with some amazing teams to help them build evals." - Positions Arize as an **evaluation tooling/platform provider** for AI teams. - **Scale claim**: > "We run over 100 million evals every month. The average team runs about 12 different eval jobs with the top teams running over 3,800 different evaluators." - Claims deep visibility into production evaluation practices across many teams. - **Product claims**: Announces a release — > "we've released agent as a judge… We've released signal. Signal's actually a long-running agent that can read traces sent in, discover patterns of issues," including claims it "can go put up a PR and put up a fix." - **Internal agent claim**: > "we have our own agent that we've built, Alex, that lives in our UI" — with claimed capabilities of long memory, dynamic UIs, and cross-trace search. ## Observable Presentation Patterns - **Audience addressed**: AI engineers, PMs, and evaluation practitioners — evidenced by > "Evals have gone from the new skill that every PM and every AI engineer has to learn" and > "the classical LLM as a judge evals, that probably many of you have written in this room." - **Claimed scope of authority**: Front-row insight into industry practice — > "We've been really fortunate to get to work with some of the best AI teams in the world… a front row seat into not just… building their actual agents… but actually the evals that teams are running on their live production agent." - **Argument style**: Problem→revelation→product structure — describes rising system complexity (2023 prompt-answering → 2024 tool calls/reasoning → sub-agents on long-horizon tasks), then positions agent-as-a-judge as the solution. - **Promotional/institutional framing**: Heavy event-level calls to action — booth visits ("We're right by the OpenAI booth"), the evals track in "room 2005," and a World Cup viewing party, indicating an **event-sponsor/organizer posture** rather than neutral educational content. - **Named-entity appeal**: Cites industry figures ("Garry Tan saying, 'Evals are everything you need'" and references to Anthropic/OpenAI CPOs) to bolster the framing that "the whole industry kind of agrees." ## Notes for Channel-Level Synthesis - The video demonstrates a **vendor-perspective talk**: technical insight is interwoven with product launch (Signal) and booth/event promotion. - Distinguish carefully: scale statistics ("100 million evals/month") are **creator claims**, not independently verifiable from the material. - Audience is clearly **practitioner/enterprise AI teams**, consistent with the channel hosting a dedicated evals conference track. # Delivery Observation: Aparna Dhinakaran — "The Future of Evals" ## Overall Tone & Register - **Conversational and enthusiastic**: opens warmly ("Well, hey everyone") and closes with an upbeat "Awesome. Thank you all so much." - Uses casual filler and connective phrases throughout: "Um," "I mean," "kind of," "cuz," "Little bit of some stats for you guys." - Reads as a **promotional conference talk** — mixes technical framing with product marketing (booth plug, demo invitation, event logistics). ## Vocabulary & Recurring Terms - **"Evals"** dominates the vocabulary — used as shorthand for "evaluations" throughout (title, "first-gen evals," "classical evals," "the future of evals"). - Repeated contrastive pairing: **"LLM as a judge" vs. "agent as a judge"** — the conceptual spine of the talk. - Frequent emphasis words: **"fundamentally," "massive," "incredible/amazing," "really"** ("really big revelation," "really lucky," "really fortunate"). ## Recurring Phrases & Patterns - **"A fundamentally different type of problem/eval/trajectory"** — used three times: > "we actually got a fundamentally different type of problem" > "That's a fundamentally different trajectory" > "you need a fundamentally different type of eval" - **Time-based framing**: structures the argument chronologically — "In 2023, it was about just answering a prompt. In 2024, we started to see..." - **Superlatives for scale/credibility**: "some of the best AI teams in the world," "amazing teams," "amazing speakers," "an enormous volume of traces." ## Teaching / Argumentative Habits - **Statistics as authority**: cites usage numbers early — "over 100 million evals every month," "about 12 different eval jobs," "over 3,800 different evaluators." - **Name-dropping for credibility**: "all the CPOs of Anthropic, OpenAI... GDB... Garry Tan saying, 'Evals are everything you need.'" (Note: the quote's wording as delivered is ambiguous — "everything you need" may be a misstatement of a known phrase; reported verbatim from transcript.) - **Self-referential case study**: uses her own product's agent ("Alex") as the failure example — "we get our kind of get to feel this pain ourselves." - **Rhetorical question pivot**: "So, we added evals, they catch all the failures, right? Here's the problem." - **Concession-then-redirect**: doesn't fully dismiss prior methods — "Doesn't mean that all of the ways that we did evals... doesn't matter anymore, but..." ## Audience Engagement - Direct audience acknowledgment: "that probably many of you have written in this room." - Explicit calls to action at the close: visit the booth ("We're right by the OpenAI booth"), attend room 2005, register via Luma for a World Cup viewing party. ## Uncertainty Notes - This is a **single short talk (~3.5 min)**; patterns like the three-part "fundamentally different" repetition are observable here but may not reflect stable habits across other talks. - Attributed quotes (Garry Tan) are delivered secondhand and cannot be verified from this transcript alone. # Observation Record: "The Future of Evals: From LLM as a Judge to Agent as a Judge" **Speaker:** Aparna Dhinakaran, Co-founder of Arize AI (self-identified in transcript) --- ## Context & Framing - Dhinakaran frames evals as having shifted from **"the new skill that every PM and every AI engineer has to learn"** to **"the thing that every serious AI team is betting on."** - Claims Arize has a "front row seat" into AI teams' pre-ship and production evals through customer work. - Talk focus: teams running evals **on traces** of live production agents. ## Stated Statistics (Arize's own platform claims) - **100+ million evals** run monthly. - Average team runs about **12 eval jobs**; top teams run **over 3,800 evaluators**. - No methodology or source provided beyond speaker's own account. ## Industry Consensus Claim - Dhinakaran asserts "the industry kind of agrees" on the importance of evals, citing CPOs of **Anthropic** and **OpenAI**, **GDB**, and quoting **Garry Tan**: "Evals are everything you need." ## Core Argument: The Evaluation Target Has Changed - **2023:** evaluation target was "just answering a prompt." - **2024 onward:** frontier models added tool calls, reasoning, and deep research; now teams run **loops on real-world data with sub-agents on long-horizon tasks**. - Claim: this was not just a harder problem but **"a fundamentally different type of problem"** — failure modes changed as systems grew more complex. ### Arize's Internal Experience ("Alex" Agent) - Alex is Arize's own UI agent with long memory, dynamic UI creation, and trace search. - Acknowledged failures: **forgetting context**, not knowing when something was done, **getting stuck in loops**. - Claim: classical **LLM-as-a-judge** evals "just weren't" sufficient to catch these failure types — a deterministic flow differs fundamentally from an agent producing a **new trajectory/UI every user interaction**. ## The Proposal: Agent as a Judge - **Revelation:** "What if the best way to evaluate an agent was actually with an agent." - Qualification preserved: this **does not** invalidate deterministic evals or LLM-as-a-judge — "we have a different type of tool to solve a different type of problem." - Contrast drawn: - **LLM as a judge:** fixed rubric, fixed scores. - **Agent as a judge:** adaptive, dynamic analysis suited to variable trajectories. - Speaker's stated position ("my take"): most teams today do the first two approaches; the future is having **all three**. ## Product Announcement: Signal - Announced release of **Signal**, described as a **long-running agent** that: - Reads submitted traces and **discovers patterns of issues**. - Detects problems classical LLM-as-a-judge evals "would never be able to do" with deterministic rubrics. - Example subtle failures cited: a **loop repeating**, a tool called **repeatedly for a long time**, an **inefficient trajectory**. - Can, from its analysis, **open a PR with a fix**. ## Promotional Close - Invites attendees to Arize's booth (near the OpenAI booth), the evals track in room 2005, and a USA World Cup viewing party. --- ## Unresolved Tensions / Qualifications - Statistics are speaker-reported platform figures with no external verification offered in the talk. - Speaker explicitly hedges that agent-as-a-judge **complements** rather than replaces prior eval methods, while simultaneously claiming classical evals "just weren't" sufficient for agent failure modes — a tension she addresses but does not fully resolve. - Effectiveness claims for Signal (e.g., catching "very subtle failures") are asserted without presented evidence in the transcript. # Speaker Attribution Record ## Video **The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI** ## Speakers ### Aparna Dhinakaran — **Creator / Presenter** - Self-identifies: > "My name is Aparna, one of the founders of Arize." - Sole speaker throughout; talk opens and closes with her remarks, followed by music. ## Roles Summary | Speaker | Role | Evidence | |---|---|---| | Aparna Dhinakaran | Creator/founder of Arize, keynote presenter | Explicit self-introduction | No guests, narrators, or other identified speakers appear in the transcript. ## Referenced Third Parties & Claims - **Industry figures quoted (claim attribution only):** CPOs of **Anthropic** and **OpenAI**, **GDB**, and **Garry Tan** — cited as saying > "Evals are everything you need." No further sourcing provided in transcript. - **Named organizations** (mentioned as partners, speakers, or benchmarks; no claims evaluated): **Term Bench**, **Uber**, **Snorkel**, **OpenAI** (booth location reference). ## Promotional Segments (Flagged) - **Arize product promotion:** Announcement of **"Signal,"** described as a released "agent as a judge" long-running agent — a product announcement embedded in the talk. - **Booth/demo solicitation:** > "come to our booth. We're right by the OpenAI booth. We'll give you a demo." - **Event promotion:** Repeated invites to the evals track in **room 2005**. - **Event promotion:** Viewing party for the USA World Cup game, with Luma registration link mentioned. ## Provenance Facts & Statistics (speaker-reported, unverified) - 100 million+ evals run monthly (Arize claim) - Average team runs ~12 eval jobs; top teams run 3,800+ evaluators (Arize claim) - Existence of internal agent **"Alex"** living in the Arize UI (speaker-reported) ## Unavailable Evidence Channels - **Comment availability:** No comment data supplied in source material. - **Curated metadata:** None provided beyond the title. - **Timestamps:** None present in transcript. \>\> Awesome. Well, hey everyone. My name is Aparna, one of the founders of Arize. We work with some amazing teams to help them build evals. Um, and we have an incredible lineup of talks for you all today at the evals track. Um, it's happening in room 2005 and there's going to be amazing speakers from Term Bench and Uber and Snorkel kind of all happening after this. Um, but today I'm here to talk to you about the future of evals. Evals have gone from the new skill that every PM and every AI engineer has to learn to the thing that every serious AI team is betting on. We've been really fortunate to get to work with some of the best AI teams in the world. So, we get a front row seat into not just what's happening when they're building their actual agents and before they actually ship, but actually the evals that teams are running on their live production agent via their traces. Little bit of some stats for you guys. We run over 100 million evals every month. The average team runs about 12 different eval jobs with the top teams running over 3,800 different evaluators. And offline evals, online evals, they each have their own place, but today what I'm actually going to talk to you about is the teams that are running evals on their traces. This is actually what's helping teams figure out what's working, catch their failures, and that's the type of data you need to fuel your continual learning loops. And the industry kind of agrees. I mean, all the CPOs of Anthropic, OpenAI, all you know, GDB, you have Garry Tan saying, "Evals are everything you need." And the whole industry kind of agrees. So, we added evals, they catch all the failures, right? Here's the problem. When we were building all of these first-gen evals, the thing that we were actually evaluating has changed underneath us. In 2023, it was about just answering a prompt. In 2024, we started to see all the frontier models. They've added tool calls, they've added reasoning, they've added deep research. Now, what we have is teams running loops on real-world data with sub-agents kicked off on long-horizon tasks. Every one of these was actually a massive jump in complexity, and we didn't just make the problem harder, we actually got a fundamentally different type of problem. What that meant is that as these systems got more complex, so did the way that they actually fail. We're really lucky cuz we have our own agent that we've built, Alex, that lives in our UI, and we get our kind of get to feel this pain ourselves. Every time the frontier labs added new functionality, we added it to our agent. And now Alex can has much longer memory. It has the ability to create dynamic UIs. It can go search across an enormous volume of traces. But, we also realized that it would forget context. It wouldn't know when something was done. Um sometimes it would just get stuck in these loops. And the key thing here is that the classical LLM as a judge evals, that probably many of you have written in this room, just weren't for us to be able to catch all the types of failures that we were experiencing. I mean, it's just fundamentally different, right? You have a deterministic flow, and now what we have is literally every time a user interacted with Alex, it would create a new UI. That's a fundamentally different trajectory. So, this led to our really big revelation. What if the best way to an evaluate an agent was actually with an agent. Doesn't mean that all of the ways that we did evals, with deterministic evals, with LLM as a judge, classic evals, doesn't matter anymore, but it just means that we have a different type of tool to solve a different type of problem. Agent as a judge is about adaptive dynamic analysis. LLM as a judge just gives you a fixed rubric with these fixed scores. It's what everyone's doing, but when your agent's doing completely different trajectories every time a user puts in data, it just means that you need a fundamentally different type of eval. My take is that most teams today are doing the first two, but the future of evals is actually having all three. And today I'm actually excited to share we've released agent as a judge to help our teams on their eval journey. We've released signal. Signal's actually a long-running agent that can read traces sent in, discover patterns of issues. Um, it can figure out types of problems that a classical LLM as a judge eval just would never be able to do with these deterministic rubrics. It's helped us figure out very subtle failures that you wouldn't even think of doing, such as something going on in a loop for multiple times, it was calling the same tool for repeatedly long time, the trajectory was inefficient. And actually what this does is because it has all that analysis, it can go put up a PR and put up a fix. So, if you want to learn more, come to our come to our booth. We're right by the OpenAI booth. We'll give you a demo, we'll show you a bit more about it. We're also, like I said, taking over the evals track, so come to room 2005. We're going to be talking a lot about the future of evals and what they look like. And if you just want to hang out with our team, we're throwing a viewing party for the USA World Cup game tonight, so check out the Luma and register to come join us. Awesome. Thank you all so much. \>\> \[music\]