Talk by Vincent Chen of Snorkel AI on the art and science of benchmarking AI agents. Motivated by the ARC AGI 3 launch (human-solvable tasks with frontier models scoring under 1%), Chen argues that AI measurement lags AI building, and that influential benchmarks are bets on future capabilities rather than snapshots of current ones. He presents a framework drawn from reviewing over 120 applications to Snorkel's $3 million Open Benchmarks Grants. The "science" covers task quality, distributional diversity, model headroom, and robust evaluation methodology; the "art" involves having a thesis (e.g., Terminal Bench's bet on the CLI before coding agents made it obvious), producing research roadmaps, and prioritizing researcher UX. He closes by identifying three axes for next-generation benchmarks: environment complexity, autonomy horizon, and output complexity beyond plain text. # Channel Presentation Observations — "The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI" ## Speaker & Channel Positioning - The speaker introduces himself as **Vincent Chen, "research fellow and a co-founder at Snorkel AI"** — an explicit self-description anchoring the talk in a company-affiliated perspective. - He describes Snorkel as a **"frontier AI data lab"** with academic ties: *"our co-founders have labs at Stanford, U Dub, Wisconsin."* - Claimed company scope: *"building the best data sets and environments to define and advance future AI capabilities."* ## Explicit Self-Description vs. Demonstrated Content ### Creator claims (stated by the speaker) - Snorkel works at the intersection of academia, frontier labs, and enterprise deployments: *"we get a lot of exposure to both the academic frontier... and also real enterprises."* - Snorkel runs an **Open Benchmarks Grant**: *"We recently... deployed $3 million to commit to open benchmarks"* and has *"reviewed I think over 120 applications."* - The talk frames a specific thesis: agent **evaluation capability lags behind agent capability** — *"our ability to actually measure these agents in practice is falling behind of where the capabilities actually are."* ### Demonstrated presentation (observable in the material) - The channel presents **conference-style technical talks** with a live audience: *"I'm very excited to be here," "every person in this room."* - The content is a **structured framework talk**: four "science" axes (task quality, distributional diversity, difficulty/model headroom, robust eval methodology) and three "art" differentiators (thesis, roadmapping, researcher UX). - Heavy use of **named benchmark examples as evidence**: GPQA, MMLU, ARC-AGI 2/3, τ-Bench, Terminal-Bench, SWE-Bench, HELM. - The speaker proposes a forward-looking roadmap of three axes: **environment complexity, autonomy horizon, output complexity**. ## Intended Audience - Primarily **AI researchers and benchmark builders**: *"built for this audience, right, a researcher audience, a builder audience."* - Includes **potential grant applicants**: a direct call to action — *"we are still accepting benchmarks in the open benchmarks grant... reach us at benchmarks.squirrel.ai."* - The framing assumes familiarity with frontier AI topics (reasoning models, coding agents, RL reward signals) — e.g., unexplained references to *"01 style reasoning"* and *"vibe coding."* ## Claimed Scope of the Channel/Content - **Topic scope**: AI evaluation and benchmarking methodology, specifically for agents. - **Positioning claim**: benchmarks should *"not just measure progress looking backwards, but really shape where things are going."* - **Evidence-sharing scope**: the talk claims to distill practical learnings — *"share some of the learnings that we've had over the last few years on what really makes for benchmarks that shape the field."* ## Notable Presentation Patterns - **Promotional-informative blend**: substantive technical content coexists with company promotion (Snorkel's grant, deployments, funding amounts). - **Accessible, conversational delivery**: humor and cultural references (*"pull a Timothy Chalamet and honor some of the greats"*), casual asides (*"maybe this is a little too American"*). - **Community-engagement posture**: repeated invitations to connect — *"please find me afterwards," "reach me directly."* ## Summary of Channel Identity Signals | Dimension | Evidence | |---|---| | **Who** | Snorkel AI co-founder/research fellow speaking at an AI conference | | **Audience** | Researchers, benchmark builders, grant applicants | | **Scope** | Agent evaluation, benchmark design frameworks, open benchmark funding | | **Tone** | Enthusiastic, framework-oriented, mixing technical depth with company advocacy | # Delivery Observation — Vincent Chen, "The Art & Science of Benchmarking Agents" ## Tone & Overall Register - **Conversational, upbeat, enthusiasm-forward**: Frequent use of "excited" and related expressions throughout the talk: - > "I'm very excited to be here" - > "we're really excited to see where the field is going" - > "I'm super excited to chat with you all" - **Warm, informal opening** directed at the room: - > "Everybody, how's it going? Lovely." - **Semi-formal conference talk register** with casual asides and self-deprecation: - > "I'm a little admittedly behind on some some reviews" - > "maybe this is a little too American, but to pull a Timothy Chalamet" ## Filler & Disfluency Patterns - Heavy, recurring filler — especially **"you know"**, "um," "uh," and **"right?"** as a rhetorical anchor: - > "closing that gap, that evaluation gap, we believe requires a toolkit, right?" - > "As I mentioned earlier, we're strong believers in field deployments, right?" - > "and and the the science, if you will" - **Repeated-word stutters** occur often at the start of clauses: - > "we see it at every person in this room, I'm sure has has played with these these these agents" - > "And And And as a result" ## Vocabulary & Recurring Phrases - **Signature repeated metaphor — "hill climbing"**: - > "we see a real progress marked by, you know, hill climbing on model cards" - > "where are all the labs going to hill climb next?" - > "how easy is it for the community to adopt and eventually kind of hill climb against this?" - **"Shape the field" / "move it forward"** as a recurring refrain: - > "benchmarks that shape the field and and move it forward" - > "not just measure, you know, progress looking backwards, but really shape where where things are going moving forward" - **Recurring vocabulary cluster**: "frontier," "real-world," "intentional," "hallmarks," "rigorous," "capabilities" — used repeatedly across sections: - > "one of the the hallmarks of great benchmarks in our view" - > "one of the hallmarks that matters when you're building these frontier evals" - **"Contact reality"** — a distinctive Snorkel-specific phrase used twice: - > "deploying engineers, researchers on our teams to again, contact reality" - > "this unique intersection of both, you know, the frontier academically, but also to contact reality with our deployments" - **Informal slang**: "vibes are improving," "special sauce," "a wild reception," "no-go at the end of the day," "pan out" ## Teaching / Structural Habits - **Explicit enumerated frameworks**: Consistently announces structure ("two themes," "threefold," "the third axis"): - > "two themes here again on the science side... and on the art side" - > "the axes for the next great benchmarks are threefold" - > "So, the first thing I want to talk about is individual task quality. Right..." - **Concrete example per principle**: Nearly every abstract point is grounded with a named benchmark (GPQA, MMLU, ARC-AGI 2, τ-Bench, Terminal Bench, SWE-bench, HELM) and a specific anecdote: - > "a model that books the right flight, but violates fair class rules, still fails" - > "ARC-AGI 3 and again at launch they had frontier models under 1%" - **Rhetorical questions to frame sections**: - > "How do you actually build good empirical measuring sticks that are actually useful?" - > "What are the new ways of coding look like?" - **Mini-concrete scenarios** to illustrate abstract categories (self-driving yellow lights; customer-experience agents losing context; org-specific codebase policies). - **Signposting transitions**: "So, let's start with the science," "I want to shift a little bit now to the differentiators," "Now, before I wrap up." ## Closing Behavior - Ends with an explicit **call to action** tied to the grant program and contact info: - > "we are still accepting, you know, benchmarks in in the open benchmarks grant. So if you're excited, uh please reach us at benchmarks.squirrel.ai" - Restates the central thesis in the final sentence ("not just measure progress looking backwards, but really shape where things are going") — consistent with his repeated framing throughout. ## Uncertainty Notes - Timestamps are only available as line markers from the transcript; no delivery pacing, pauses, gestures, or audience reaction are observable from text alone. - The transcript contains a few transcription irregularities (e.g., "Tow Bench" for τ-Bench, "SweetBench" for SWE-bench, "benchmarks.squirrel.ai" likely meaning snorkel.ai); these are noted as-is without correction beyond speaker intent. # Observation Document: The Art & Science of Benchmarking Agents **Speaker:** Vincent Chen — Research Fellow and Co-founder, Snorkel AI **Source:** Video transcript, "The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI" ## Framing: The Evaluation Gap - Chen identifies an **asymmetry** between agent capability and measurement: real excitement and progress ("hill climbing on model cards," "the vibes are improving," especially in coding), yet individuals and enterprises hesitate to deploy agents in **high-stakes environments**. - His qualification: "that's not to say the capabilities aren't there, but our ability to actually measure these agents in practice that is falling behind." - Closing this **evaluation gap** is described as one of the most important research questions in the field. - Snorkel's position: closing the gap requires a toolkit — field deployments (finance, insurance, healthcare), red teaming, private human evals, crowdsource labeling — but **open benchmarks remain critical**. - Claim: the best open benchmarks "aren't just about taking a snapshot of progress looking backwards. They're actually about defining progress and shaping the field." Cites Terminal Bench, METR's Long Horizon Benchmark, and ARC-AGI as recent guideposts. ## Context: Snorkel AI and the Open Benchmarks Grant - Snorkel described as a "frontier AI data lab" with academic co-founder labs (Stanford, UW, Wisconsin) and enterprise deployment exposure. - Recently deployed **$3 million** toward open benchmarks; over **120 applications** reviewed so far, spanning academia and industry labs. ## The "Science": Effective Measuring Sticks ### 1. Individual Task Quality - Tasks must be rigorously validated, represent real-world complexity, have well-posed instructions and verifiable solutions validated by domain experts. - **Example — GPQA:** Chen highlights its appendix: a **multi-reviewer adversarial quality control protocol** (original author, reviewers, adjudicators, revision opportunities) and **incentive mechanisms** where payouts depended on agreement. He acknowledges inspiration from peer review, "as flawed as that is." ### 2. Distributional Diversity - Define a clear taxonomy for the domain and distribute tasks intentionally — either to represent real traffic traces or to deliberately cover **rare-but-disproportionately-important failure modes** (e.g., yellow lights, pedestrians, motorcyclists in self-driving). - **Example — MMLU:** its ambitious 57-domain taxonomy is credited by Chen for its longevity. ### 3. Difficulty and Model Headroom - Benchmarks should be **unsaturated**, expose soft spots, and reliably separate frontier models. - **Example — ARC-AGI 2:** designed for a capability humans have and models lack; remained unsaturated, then the reasoning push ("o1-style," ~18–24 months prior) produced a leap correlating with benchmark performance. **ARC-AGI 3**, launched weeks earlier, had frontier models **under 1%** at launch, with every task human-solvable. ### 4. Robust Eval Methodology - Go beyond accuracy: capture cost, latency, reasoning trace quality, intermediate steps, tool use as reward/supervision signals. - **Example — Tau Bench:** evaluated multi-turn task completion via a user simulator plus **adherence to policy constraints**; a model booking the correct flight while violating fare-class rules still fails. ## The "Art": Differentiators of Frontier-Shaping Benchmarks ### 1. A Thesis - Great benchmarks embody a research question or bet about where the field is going. - **Example — Terminal Bench:** a bet on the CLI as core interface for general-purpose computer use, not just coding. Chen argues this "largely correct and consequential bet" helped accelerate how the field operates. ### 2. Roadmapping for the Field - Great benchmarks inspire new research attacks and benchmark families. - **Example — SWE-bench:** simple idea leveraging PRs; spawned a family (Light, Verified, Pro, Multilingual, Multimodal). Chen claims there's more room to innovate (e.g., "vibe coding"). ### 3. Researcher UX (described as "severely underrated") - Easy to run models against, easy to contribute/extend tasks, easy to use signals for RL or post-training. - **Examples:** Stanford CRFM's **Holm [HELM]** — standardized, modular harness (Chen credits it as pioneering this); **Terminal Bench 2.0's Harbor** — de facto harness for agent evaluation infrastructure. ## Proposed Next-Wave Benchmark Axes (Snorkel's internal roadmap) ### 1. Environment Complexity - Real environments include org-specific policies, Slack context, screenshots, flaky toolchains, distributed CI, human reviewer preferences, parallel contributors — "benchmarks today capture a fraction of this complexity." ### 2. Autonomy Horizon - How long agents operate before reliability breaks down; capturing points on the autonomy slider (co-pilot vs. fully autonomous). Example: customer experience agents losing context delivered weeks ago, changing product specs, reorgs shifting priorities midstream. ### 3. Output Complexity - Under-explored; beyond chat/document outputs toward nuanced reward signals, real artifacts, and "new types of form factors that we haven't even imagined" (agent-human and agent-agent interaction). Includes **trustworthy outputs** — agents capturing their own uncertainty and asking for more information. ## Call to Action - Snorkel is still accepting applications to the Open Benchmarks Grant (contact: benchmarks.squirrel.ai / reach Chen directly). - Closing framing: benchmarks should "not just measure progress looking backwards, but really shape where things are going moving forward." ## Unresolved Tensions Noted - Capability vs. measurement gap: capabilities exist but enterprise hesitation persists — attributed by Chen to measurement falling behind, not capability absence. - Chen frames the "art" section as differentiators while acknowledging the "science" examples also pushed the frontier — an acknowledged overlap, not a contradiction. - Minor transcript ambiguity: "Stanford team at CRFM built Helm" appears as a slight disfluency in naming; recorded as expressed. # Speaker & Provenance Observation — Video 01a15ca9-1353-7961-b539-20422abb8e4f ## Video **Title:** The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Vincent Chen** | **Creator / Presenter** (sole speaker) | Self-introduction: "my name is Vincent. I'm a research fellow and a co-founder at Snorkel AI" | - No guests, narrators, or other identified speakers appear in the transcript. - An **unseen audience** is addressed ("Everybody, how's it going?") but does not speak on-record. ## Promotional Segments - **Snorkel AI company description** — "a frontier AI data lab"; co-founders' labs at Stanford, U Dub (UW), Wisconsin. - **Open Benchmarks Grant promotion** — "$3 million to commit to open benchmarks"; explicit call-to-action near the end: submit via **benchmarks.squirrel.ai** (as spoken) and "reach me directly." - These segments are organizational promotion delivered by the speaker and are flagged as such. ## Referenced Sources & Third-Party Claims Claims about third-party benchmarks are **speaker characterizations**, not verified facts: - **GPQA** — graduate/professional knowledge benchmark; appendix cited for adversarial quality-control (multi-reviewer protocol, incentive-based payouts). - **MMLU** — 57-domain taxonomy benchmark. - **ARC-AGI 2 / ARC-AGI 3** — ARC Prize Foundation credited; claim that frontier models scored "under 1%" at ARC-AGI-3 launch. - **Tau Bench ("Tow Bench")** — multi-turn agent eval with user simulator and policy-constraint checking (flight-booking example). - **Terminal Bench / Terminal Bench 2.0 (Harbor harness)** — described as a CLI/computer-use bet that "panned out." - **SWE-bench** family (light, verified, pro, multilingual, multimodal). - **HELM** — credited to "the Stanford team at CRFM." - **Metr's Long Horizon Benchmark, "ARK GI"** — named as recent guidepost benchmarks (as transcribed). - Referenced eval methods: red teaming, private human evals, crowdsourced labeling, peer review (as an inspiration analogy). ## Organizational Affiliations (per speaker) - Snorkel AI (employer/co-founder) - Open Benchmarks Grant program (speaker role: reviewer, "over 120 applications") - Academic partners: Stanford, University of Washington, University of Wisconsin (labs said to belong to co-founders) ## Comment Availability - **Not indicated** — no metadata on comment availability was supplied. ## Uncertainty Notes - Transcript contains speech disfluencies; "ARK GI," "Tow Bench," "SweBench/SweetBench," and "benchmarks.squirrel.ai" are rendered as spoken and may be transcription artifacts. - Statistics (e.g., "$3 million," "120 applications," "under 1%") are speaker-stated figures with no independent verification in the source. Everybody, how's it going? Lovely. Well, I'm very excited to be here hailing from San Francisco. It's a little bit of a trek over, but I'm super excited to to uh chat with you all um today after the talk uh and beyond. Uh my name is Vincent. I'm a research fellow and a co-founder at Snorkel AI. And uh you know, today I'm going to be talking about um some meta evaluations for building benchmarks, the art and science of what we found to be really useful when building effective benchmarks. I have the great privilege at Snorkel of working with both our researchers, uh great collaborators in academia, industry, uh and the open source community to build great benchmarks. And I wanted to share some of the learnings that we've had over the, you know, last few years on on what really makes for benchmarks that uh shape the field and and move it forward. So, a little bit about us. Uh we're a frontier AI data lab. Uh we have, you know, labs both at uh at academic settings, you know, our our co-founders uh have have labs at Stanford, U Dub, Wisconsin. We've an internal team uh for deployed engineers, applied engineers. And uh I mention this because we get a lot of exposure to both, you know, the academic frontier, work with frontier labs, and also real enterprises and and companies who are deploying in practice. So, our focus as a company is on building the best data sets and environments to define and advance future AI capabilities. And we are in a fortunate spot where we get to play at this unique intersection of both, you know, the frontier academically, but also um uh uh to contact reality with our deployments um in enterprises. So, today I wanted to talk about an asymmetry that we see in our real-world deployments. There's real excitement around agents today. You know, we see it at every person in this room, I'm sure has has played with these these these agents and we see a real progress marked by, you know, hill climbing on model cards. We see the vibes are improving, right? And truly shifting, especially in coding. But when you ask individuals and enterprises or these, you know, large-scale organizations if they're fully ready to let these agents loose and you know, deploy them in high-stakes environments, you get a little bit of hesitation. And that's not to say the capabilities aren't there, but our ability to actually measure these agents in practice that is falling behind of where the capabilities actually are. This is one of the challenges and research questions that I think are actually one of the most important in the field and one of the ones that we're very interested in here at Snorkel. So, closing that gap, that evaluation gap, we believe requires a toolkit, right? As I mentioned earlier, we're strong believers in field deployments, right? So, this is you know, actually deploying engineers, researchers on our teams to again, contact reality and and and work with folks to deploy these models in real production settings where the stakes are high, right? These are finance settings, insurance, you know, healthcare settings where it's not just about a number, it's about real outcomes. And we're also big fans of other eval tools, right? This is red teaming, private human evals, crowdsource labeling, a lot of the the themes that we saw talked about today, which which has been awesome to see. But one of the things that we feel most strongly about is that benchmarks, open benchmarks in particular, remain a really critical piece of the measurement toolkit. The best open benchmarks aren't just about, you know, taking a snapshot of progress looking backwards. They're actually about defining progress and shaping the field and setting a goalpost about where capabilities need to go. And you know, even looking at the last few months, right? Benchmarks like Terminal Bench, uh Meters Long Horizon Benchmark, uh ARK GI, these are really exciting and critical guideposts for where the field is going. And And And as a result, the path to safe, trustworthy agents um will will really depend on more of these benchmarks in practice. So, what are we doing at Snorkel? One of the things that again I'm I'm very fortunate to to be able to be a part of is uh the Open Benchmarks Grant. We recently, a few weeks ago, a month ago, uh deployed $3 million to commit to open benchmarks. And this is really fun job. I get to work with the best academic teams, you know, builders uh to really accelerate and fund um the next wave of benchmarks that's going to really steer and and and guide where uh the field is going. We've had a wild reception so far. I'm a little admittedly behind on some some reviews. Uh but we've been really excited to see what the community has come up with so far. And in this talk in particular, you know, we've reviewed I think over 120 applications so far, spanning academia and industry labs. We wanted to share a few perspectives, a few learnings um over the fast past few months about what we view as one table stakes for for useful benchmarks, right? How do you actually build good empirical measuring sticks that are actually useful, you know, to to measuring progress? And two, what really separates, you know, those benchmarks that are shaping the frontier, right? What it What is the art and and the the the science, if you will, of building really official effective benchmarks at the end of the day. So, as I'm doing this, I'll have a fun opportunity, uh maybe this is a little too American, but to pull a Timothy Chalamet and uh honor some of the greats, you know, some of the great benchmarks of the last few years uh that have really shaped the field um in talking about some of these axes. And I hope that these, you know, themes resonate with you and also inspire a bit of um you know, kind of new thinking about, "Hey, how how can we actually deploy some of the learnings we're we're all kind of driving towards in our day-to-day work to, you know, shape the field and and move it forward. So, two themes here again on the science side, you know, how do we actually build effective measuring sticks? We'll talk about task quality, distributional control, uh robust evals in general. Um and on the on the art side, right, really the the differentiators for great benchmarks, um how do you build benchmarks with a thesis on where the field is going that inspire new road maps, um and that critically are are built for this audience, right, a researcher audience, a builder audience so that adoption is is something that is a way smoother um and and a first-class citizen uh for a bunch of these benchmarks. So, let's start with the science. Uh this is again what makes for really effective measuring sticks as we've seen them in in practice and in the the deployments that we see uh in industry and academia and with frontier labs. So, the first thing I want to talk about is individual task quality. Right, this is the idea that individual tasks need to be exceptionally rigorously validated, right? They need to represent real-world complexity. They need uh well-posed, well-structured instructions. Um they need verifiable solutions uh that ideally have been um actually validated by real-world domain experts. Um one of the the benchmarks here, GPQA, is one of my favorites um not just because it's been a very lasting and enduring benchmark that captures, you know, uh uh graduate-level and professional knowledge uh even to this day, right? You kind of still still see this on model cards. But, one of my favorite contributions is actually tucked away in the appendix. GPQA uh introduced one new adversarial qual- uh quality control mechanism. So, the idea was that not only do these tasks need to be well-posed, um they need to be tractable for other experts to solve. So, they had a very very rigorous multi-reviewer protocol where there was an original author, you know, there were there were reviewers and adjudicators in the loop. There was opportunity for revision, right? These were tasks that were really pushing the frontier of knowledge and it was non-trivial for any single expert to say, "Yeah, this is actually a good task or not." And so, developing this sort of rigorous adversarial quality control mechanism was one of the contributions I was most excited about here. And if you read the appendix, you also see that they introduce new incentive mechanisms, right? Payouts are actually based on whether there was certain agreement and, you know, uh uh coming from academia, you know, there's there's uh some inspiration here from the peer review process as as flawed as that is, but, you know, this type of innovation around how you actually get really rigorous, you know, multi-expert quality control um leads to the type of outcomes that we see around individual task quality that um we we see as a a key foundation for any benchmark that matters at the end of the day. Two is uh distributional diversity, right? This is the idea that for any benchmark that really matters, you want to define a clear taxonomy for the domain, for real-world tasks, and distribute those tasks uh intentionally. So, this might be, "Hey, I I captured a trace or or kind of real-world stream of um traffic along um you know, how my agent is operating in the real world, and I want to really represent that distribution." It could also mean, "Hey, I'm specifically characterizing and taxonomizing the failure modes that are, you know, paradoxically rare, but you know, disproportionately important in in production, right? If you take classic self-driving settings, right? Yellow lights or, you know, pedestrians or uh motorcyclists, right? Might actually show up way less than other other types of scenarios, but are disproportionately important, you know, to get right in these settings. And so, defining that taxonomy, being really intentional about distributing tasks across it is one of the hall hallmarks of uh great benchmarks in our view. Um MMLU, um few years old now, uh constructed a a quite ambitious taxonomy of, you know, 57 academic and and professional domains across STEM, humanities, et cetera. It's remained one of the lasting benchmarks for understanding graduate and professional level knowledge and again a lot of this was as as we we believe a result of really thoughtful and intentional taxonomy designed and and building towards that. The third axis here is around difficulty of individual tasks and model headroom. Right? It's really important that the benchmark is unsaturated that it exposes real soft spots in capabilities and reliably separates where models sit at at the frontier. One of my favorite plots is is the one on the top right. This was you know all all credit to the ARC Prize Foundation team. ARC-AGI 2 you know for a very long time was unsaturated right for for several months and years. And when there was the big reasoning push you know maybe 18-24 months ago we saw a massive leap in capabilities that actually corresponded to a real leap in model capabilities. Right? This was a benchmark that was intentionally designed to represent a type of efficiency or capability that humans have but models didn't have and and they really kind of captured well hey there's a lot of model headroom here. Humans can do this. Where's that gap? And again low and behold it correlated quite well with the recent you know 01 style reasoning push that has really dominated the field in the in the past 18-24 months. Just a few weeks ago the ARC team just launched AGI ARC-AGI 3 and again at launch they had frontier models under 1%. Every single task was a human solvable to some degree and so it remains one of the I think the the most meaningfully exciting benchmarks in the space where you know any new model you know people are kind of awaiting hey how does it do do on ARC? And and I think they did this quite well. Right? The the kind of model headroom here is is really really exciting. This last axis here I want to talk about on on the empirical measurement side is all about a robust eval methodologies. Now this goes really deep. So just kind of capturing some of the high-level ideas, benchmarks need to ideally go beyond accuracy to capture real-world dimensions that matter, right? This is everything from cost, latency, you know, the quality of the reasoning traces, uh uh some of the intermediate steps and and and tool use. Whatever dimensions actually matter for the capability at hand, uh capturing those as reward or supervision signals is really critical and measuring what it what it claims to um is is actually a non-trivial uh feat uh in in uh you know, building robust and and reproducible benchmarks. So Tow Bench is a benchmark that we're we're a big fan of, you know, it's had multiple evolutions over the year, but over the years, but it would it was a benchmark that was built to evaluate both task completion of these multi-turn agents. They built a a clever, you know, kind of user simulator, um but also, you know, not just accuracy and completion, but adherence to policy constraints. So a model, for example, on the right-hand side, this was the one of the uh examples from the paper, a model that books the right flight, but violates fair class rules, still fails, is still a kind of um no-go at the end of the day. So um this notion of hey, being intentional about what axes we actually care about, uh what do we actually want to measure and measuring that rigorously, uh is one of the hallmarks that uh matters when you're when you're building um these these frontier evals. So I want to shift a little bit now to the differentiators, right? What what actually leads to the benchmarks that push the frontier. That's not to say, you know, anything I mentioned on the last few slides have not pushed the frontier. These are just special characteristics that I view as you know, critical to to the benchmarks that are real research contributions, that are really shaping where is the field going, where are the where are all the labs going to hill climb next? Um and uh this is this is the art, the special sauce that um helps push us forward. So one of the key hallmarks here is um these benchmarks should have a thesis, right? They should have a research question about a subspace of capabilities, about where the field is going. It should revisit previous capabilities. Um, and the most ambitious benchmarks are really a statement about where the world is going. Terminal bench um, is one of these bets, right? It was a bet on the CLI not just for coding agents, but for general purpose computer use. And in many ways I think this has turned out to be a largely correct and and consequential bet, right? As as we're seeing uh, teams that had kind of Claude and and and Codex build their general purpose, you know, enterprise capabilities on top of these um, coding and CLI based tools, we're seeing this bet pan out. And again, Terminal bench remains one of the most robust and kind of uh, most important benchmarks that are measured on on all the recent uh, model cards. So, again, this was a bet early on to say, "Hey, we think the CLI is going to be really important um, as a core interface, a core abstraction and affordance for agents to interact with the real world in general purpose way. And uh, by measuring those capabilities, you know, I I'd argue that it actually um, helped accelerate uh, you know, how how the field is operating in this way. The second piece here I think worth uh, mentioning is uh, the ability to kind of roadmap for the field. Um, a a great benchmark, you know, one that really shapes where where all of us are going is producing new roadmaps, right? It's inspiring new attacks against research problems. It's helping folks ideate and come up with new ways for thinking about uh, benchmarks and methods in general. And I think SweBench is uh, really phenomenal example of this, right? It was a simple idea as often the best ones are quite simple, right? Head, how do you um, kind of leverage um, you know, existing uh, coding coding type capabilities via via PRs. Uh, and it spawned a new family of benchmarks, right? all the way from SweBench light, verified, pro, multilingual, multimodal, etc. And its evolution I think is is still very relevant today. It's evolved how we think about coding agents and one of the things that's been awesome to see with the SweetBench team is how many new research directions and kind of inspired benchmarks have come after it in this coding space. And arguably I'd say, you know, there's a lot more room to kind of innovate on top of this as well. What are the new ways of coding look like? How do these types of workflows apply to 5 coding and kind of this new layer of abstraction that that's offered developers are applying? I think it's been really exciting to see the the foundation that the SweetBench team set and how that's going to shape, you know, how we think about coding agents moving forward. So, this team here I think is severely underrated and this is the notion of researcher UX. Right? I think the most prescient benchmark builders are committed to the researcher and builder experience. This is to say, it's really simple to run models and agents against your benchmark. It's really simple to contribute new tasks to extend. And also it's it's really simple to leverage some of the the signals that you're getting for the benchmark for RL or kind of tuning on post talk. I think this is really underrated. It's you know, a classic product principle to make what you're building and putting out there easy to use by the community or by your core users. And in this case, benchmarks have core users which are other builders or researchers. And so, really putting in time and attention to building those interfaces has been important for the adoption of some of the most important benchmarks. Right? To call out I think the the Stanford team at CRFM built Helm you know, several years ago which I I'd argue kind of pioneered a standardized modular harness for evaluating reproducible, you know, different scenarios as well as kind of models against a standard test bed of models. Terminal Bench 2.0 just a few months ago again shipped with Harbor which has been in many ways a de facto harness and and kind of evaluation infrastructure for teams who are building agents more broadly. And so, you know, thankfully we have a bunch of open source software out there today, you know, based on this principle. But as you're building your benchmarks, right, kind of considering, hey, how easy is this to extend? How easy is it for the community to adopt and eventually kind of hill climb against this? I think is a severely underrated factor for for what makes for really high adoption of of these frontier benchmarks. So, this is the full framework. Again, can go into more detail and and please find me afterwards. But again, what makes for really empirically meaningful uh measuring sticks, right, it's task quality and attention to distributional control and diversity. Uh it's it's difficulty and model headroom. And of course, a robust eval methodology that measures the concrete axes that actually matter and it matter in practice and is is intentional about it. And of course, on the off side, right, these these great benchmarks really have a thesis on where the frontier's going. Uh they set road maps for the field and they really prioritize high research UX. Now, before I wrap up, um I want to propose, you know, a few dimensions that uh we're really excited about at Snorkel uh that we think are really going to encapsulate, you know, the the next wave of of benchmarks. Um tried to leave some more degrees of freedom here for creativity, but these are areas where we think there's a lot of room to push complexity, to push, you know, realism in benchmarks. And so, I wanted to share a little bit of our internal road map and thinking around where the field is going and where we need more benchmarks and more contributions. So, this is our point of view. Um we think that uh the the axes for the next great benchmarks are threefold. I mean, I'll go into a little bit more detail about what I mean here um in just a second. But one, it's environment complexity, right? How complex, how realistic, how dynamic is the operating environment that these benchmarks are working? Are they representative of real-world settings that a professional, that, you know, scientist, that someone using these tools could actually use? Two is autonomy horizon. Do these benchmarks represent realistic and frontier horizon lengths that these agents are operating against? Are they capturing different points on the autonomy slider that are again representative of how users are using them as co-pilots versus fully autonomous agents? Is this an intentional design in the benchmark? And three, capturing the wide range of output complexity. I think this is very under explored today, right? Lots of chat-based or document-based outputs um not as much around nuanced uh kind of differentiated reward signals, right? Real artifacts that show up um you know, in day-to-day work that we that we represent. And critically, um new artifacts, right? New types of form factors that we haven't even imagined about you know, how agents interact with humans, how agents interact with each other. So, a little bit about each one. Um the first one here, I won't go into all of these in in significant detail, right? Environment complexity is all about capturing the real-world complexity that that is uh in our in our day-to-day working environments, right? And this this gap is often where agents fail today. Consider coding agents, right? A real codebase has org-specific policies, you know, lots of Slack context, uh screenshots, flaky toolchains, you know, CI that's that's kind of distributed, um human reviewers with knowledge in their heads about what they like and what they prefer, um many contributors in parallel. Benchmarks today capture a fraction of this complexity and you know, not just in coding but other domains, there's a lot of excitement and an opportunity to up the level of complexity and continue to drive what these models um can do to to represent real-world uses. Two, again on autonomy horizon, right? Uh this is all about how long an agent can operate before it reliably before reliability breaks down. Right, let's take a customer experience agent, um you know, in in many cases, right? Uh these these agents may lose track, you know, of of context that was, you know, delivered a few weeks ago, different integrations or product specs might, um, you know, change uh the actual spec or requirements for for a particular model. Reorgs can kind of shift, you know, priorities midstream. Um, real-world settings actually represent a lot more complexity that again is represented in these kind of long-term continual learning type type settings that represent kind of changes in state and environment. Um, so we again think that there's a lot of room to contribute and and build out new benchmarks that represent very very long horizon and an autonomous agents. And lastly, this axis is all about producing more complex work, more representative work, and also nuanced signals that can be used for not just evaluation, but reward signals during training. This also has to be complex, and this gap is is growing as well, right? Let's take again a software example, right? Or or a complex report for making strategic recommendations. It's non-trivial and subjective, you know, to define, "Hey, what is verifiable about a good recommendation, a good strategic proposal, a good roadmap in general?" Um, the nuances of this need to be captured well. They need to capture organizational context, really good human judgment, um, and tomorrow's benchmarks really, you know, we're we're excited about, um, signals that capture all of these settings. Trustworthy outputs, right? The ability for agents to actually capture their own uncertainty and uh define, "Hey, I'm actually not sure about this. I actually need to stop or kind of ask for more information." Again, different types of outputs that aren't just, you know, a kind of plain text answer is something we're really excited about. So again, uh hopefully, you know, this inspired a little bit of thinking around, "Hey, what am I working on? How can I turn this into a meaningful benchmark?" Um, we are still accepting, you know, benchmarks uh in in the open benchmarks grant. So if you're excited, uh please reach us at benchmarks.squirrel.ai. Feel free to reach me directly, and uh we're really excited to see uh where the field is going and to again to use benchmarks to not just measure, you know, progress looking backwards, but really shape where where things are going moving forward. Thanks for your time and excited to catch up soon.