--- schema: 1 type: video title: 'The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI' description: >- ARC AGI 3 launched a few weeks before this talk with every task human solvable and frontier models under 1%. That gap is the argument: our ability to measure AI has fallen behind our ability to build it, and benchmarks that actually shape the field are bets on where capabilities are going, not snapshots of where they are. Vincent Chen draws a framework from reviewing over 120 applications for Snorkel's $3 million Open Benchmarks Grants. The science is task quality, distributional diversity, model headroom, and robust eval methodology. The art is having a thesis (Terminal Bench bet on the CLI before coding agents made it obvious), producing research roadmaps, and treating researcher UX as a first class citizen. He closes on three axes he thinks the next generation of benchmarks needs to cover: environment complexity, autonomy horizon, and output complexity beyond plain text. Speaker info: - https://x.com/vincentsunnchen - https://www.linkedin.com/in/vincentsunnchen - https://github.com/vincentschen --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/iNkFlCiij0U-the-art-science-of-benchmarking-agents Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/the-art-science-of-benchmarking-agents-vincent-chen-20422abb8e4f markdown: >- https://liquidgold.cc/channel/aidotengineer/video/the-art-science-of-benchmarking-agents-vincent-chen-20422abb8e4f.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T09:51:53.794Z' language: null children: [] videoId: iNkFlCiij0U durationSeconds: 1405 publishedAt: '2026-06-04T16:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 146 endSeconds: 236 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=146' heldReason: rights - ordinal: 1 text: null startSeconds: 236 endSeconds: 247 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=236' heldReason: rights - ordinal: 2 text: null startSeconds: 247 endSeconds: 347 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=247' heldReason: rights - ordinal: 3 text: null startSeconds: 358 endSeconds: 437 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=358' heldReason: rights - ordinal: 4 text: null startSeconds: 450 endSeconds: 537 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=450' heldReason: rights - ordinal: 5 text: null startSeconds: 537 endSeconds: 547 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=537' heldReason: rights - ordinal: 6 text: null startSeconds: 547 endSeconds: 658 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=547' heldReason: rights - ordinal: 7 text: null startSeconds: 658 endSeconds: 669 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=658' heldReason: rights - ordinal: 8 text: null startSeconds: 669 endSeconds: 748 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=669' heldReason: rights - ordinal: 9 text: null startSeconds: 748 endSeconds: 759 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=748' heldReason: rights - ordinal: 10 text: null startSeconds: 759 endSeconds: 898 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=759' heldReason: rights - ordinal: 11 text: null startSeconds: 898 endSeconds: 908 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=898' heldReason: rights - ordinal: 12 text: null startSeconds: 908 endSeconds: 1020 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=908' heldReason: rights - ordinal: 13 text: null startSeconds: 1020 endSeconds: 1032 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=1020' heldReason: rights - ordinal: 14 text: null startSeconds: 1141 endSeconds: 1242 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=1141' heldReason: rights - ordinal: 15 text: null startSeconds: 1242 endSeconds: 1253 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=1242' heldReason: rights - ordinal: 16 text: null startSeconds: 1253 endSeconds: 1405 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=1253' heldReason: rights - ordinal: 17 text: null startSeconds: 1253 endSeconds: 1405 watch: 'https://liquidgold.cc/watch/iNkFlCiij0U?t=1253' heldReason: rights --- # The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI ARC AGI 3 launched a few weeks before this talk with every task human solvable and frontier models under 1%. That gap is the argument: our ability to measure AI has fallen behind our ability to build it, and benchmarks that actually shape the field are bets on where capabilities are going, not snapshots of where they are. Vincent Chen draws a framework from reviewing over 120 applications for Snorkel's $3 million Open Benchmarks Grants. The science is task quality, distributional diversity, model headroom, and robust eval methodology. The art is having a thesis (Terminal Bench bet on the CLI before coding agents made it obvious), producing research roadmaps, and treating researcher UX as a first class citizen. He closes on three axes he thinks the next generation of benchmarks needs to cover: environment complexity, autonomy horizon, and output complexity beyond plain text. Speaker info: - https://x.com/vincentsunnchen - https://www.linkedin.com/in/vincentsunnchen - https://github.com/vincentschen --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/iNkFlCiij0U-the-art-science-of-benchmarking-agents Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/the-art-science-of-benchmarking-agents-vincent-chen-20422abb8e4f · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.