Talk by Nick Heiner (Surge AI) critiquing "benchmaxxing" — the gap between benchmark scores and real model capability. He explains how chasing benchmark numbers undermines whether tests reflect reality, and outlines common benchmark failure modes: broken tasks, contamination (models memorizing test content, inflating SWE-bench-style scores with recall), reward hacking, and misalignment between prompts and verifiers (e.g., contradictory prompt constraints or parsers that can't handle answer formats, forcing gaming to score well). He recommends applying domain expertise, aligning tools with prompts, and investing in human evaluation to raise the standard for both benchmark writers and labs. # Channel Identity Observations: "When Will The Benchmaxxing Plague End?" ## Presenter & Affiliation - The speaker is **Nick Heiner of Surge AI**, per the title; within the talk he identifies his employer: *"...at Surge, one of our differentiators has long been that we are not trying to minimize cost."* - He positions himself as a **benchmark creator**: *"Both the people making the benchmarks like myself and the people who are reporting on the benchmarks."* ## Intended Audience - An **in-person technical audience with AI industry knowledge**, evidenced by the aside *"many of us in this room are looking towards AI that's about to remake entire industries"* and the conference-talk format (*"Thank you so much for coming," "a few extra slides"*). - Practitioners fluent in model names, benchmarks, and jargon (Opus, Haiku, SWE-bench, ELO/LM Arena, IFEval, reward hacking) with no definitions offered beyond "benchmaxxing" itself. - **Potential recruits**: the talk closes with an explicit hiring pitch — *"at Surge, we're hiring for basically all aspects of that."* ## Claimed Scope / Domain - **AI evaluation and benchmarking**: the talk's stated agenda is *"why does benchmaxing happen? Why are traditional benchmarks not always accurate reflections of real world value? Is this intrinsic to all benchmarks? And will we ever know which models are best?"* - Covers benchmark design anti-patterns (cost, contamination, reward hacking, verifier alignment, QC), lab-side benchmaxxing tactics, and a proposed methodology for better benchmarks. ## Self-Description Evidence - **Explicit claims by the creator:** - Surge's differentiator is **quality over cost**: *"we are not trying to minimize cost. We are trying to maximize quality... paying a lot of money for good workers."* - Surge built **Hemingway Bench**: *"we created a benchmark called Hemingway bench to measure writing"* using *"a workforce of thousands of professional writers."* - Surge conducted a contamination investigation: *"we actually did an investigation where we compared looking at the repos that SWE-bench verified was built out of."* - **Calls to action**: hiring pitch and *"if you'd like more spicy takes from me, please follow my substack"* — indicating a companion written channel. ## Observable Presentation Patterns (Demonstrated, Not Claimed) - **Critical, adversarial analysis of named industry benchmarks**: ELO/LM Arena, SWE-bench Verified, IFEval, Apex ("a rag benchmark"), with specific cited failure modes (e.g., contradictory IFEval prompts, Cyrillic-I reward hacking). - **Named-insider citations**: Andrej Karpathy, "Wor" [presumably a thought leader as transcribed], Meta's undisclosed 27-model testing — grounding claims in industry figures. - **Promotional framing embedded in technical content**: Surge's products/values are presented as solutions to the diagnosed problems. - **Skeptical-of-hype tone**: opens with *"In the tech industry, we love a hype cycle"* and critiques *"chart crime."* ## Claims vs. Demonstration Distinction - Surge's quality-over-cost positioning is a **creator claim**; the demonstration supporting it is the described Hemingway Bench methodology and contamination investigation. - Industry critiques (e.g., Opus memorizing SWE-bench, LM Arena workforce gaming) are presented as the speaker's assertions and cited anecdotes — some explicitly hearsay: *"I've heard stories of"* crowdsource-vote manipulation. - No private motivations beyond stated incentives ("marketing or... organizational politics") are asserted. # Delivery Observations: “When Will The Benchmaxxing Plague End?” — Nick Heiner ## Tone - **Conversational, dry-humored, and direct.** The speaker opens casually — “Let's get started” — and immediately delivers a joke ending: “the answers are incentives, poor methodologies, no and yes. All right, that was my talk. Thank you so much for coming. Um actually it looks like I have a few extra minutes.” - **Sardonic edge toward industry practices.** Phrases like “we do a little chart crime,” “the response is absolutely deranged,” and “for more spicy takes from me, uh please follow my substack” signal a deliberately irreverent register. ## Vocabulary & Recurring Phrases - **Neologisms and jargon used fluently:** “benchmaxing,” “chart crime,” “antiatterns” (apparent blend of “anti-pattern”), “reward hacking,” “eval awareness,” “hill climbing.” - **Repeated conceptual anchor:** the phrase **“human eval”** recurs as the standard of value — “the core value that we're all trying to get towards as human eval,” “no human eval is ever going to choose this,” “Human eval is very expensive.” - **Repeated metaphor about frontier limits:** “you can't push the frontier forward from within the frontier” appears twice — once for benchmark task creation, once as “you can't expand the frontier from within the frontier” about LLM-as-judge for writing. ## Teaching Habits & Structure - **Signposted question-driven structure.** He frames the talk with explicit questions up front: “why does benchmaxing happen? Why are traditional benchmarks not always accurate…? Is this intrinsic…? And will we ever know which models are best?” - **Definition-before-discussion pattern.** New terms are defined inline: “Benchmaxing, of course, being when labs are training too hard on benchmarks…”; “Reward hacking is basically when a model finds a lazy and creative way to meet the letter of the law, but not the spirit.” - **Concrete examples with named artifacts.** Nearly every claim is grounded in a specific benchmark (SWE-Bench Verified, Arena (EloLMArena), IFEval, Apex, AutomationBench) or a named figure (Andrej Karpathy). - **Worked cost arithmetic.** He walks through explicit numbers: “a thousand tasks… 60 hours to make… half a million a year. That's $15 million… that's $5 million to replace them.” - **Contrastive illustration.** A memorable pair: “Haiku scores 20% because it makes a bunch of mistakes and Fable scores 20% because it gets it right 80% of the time but then just happens to pick different formats.” ## Rhetorical Patterns - **Analogies for technical concepts.** “Gradient descent is basically like water flowing downhill looking for the path of least resistance.” Benchmarks as “our lighthouse on the horizon.” - **Prescriptive checklist near the end:** “start with great human experts,” “you need high fidelity input data,” “The tools need to actually work,” “You need verifiers that are fully aligned with the prompts.” - **Self-inclusion to build credibility:** “even myself, right? Like unless I actually look at a benchmark in a fair amount of detail, I don't have an opinion on it”; “Both the people making the benchmarks like myself…” - **Employer framing repeated:** “At Surge, one of our differentiators has long been that we are not trying to minimize cost. We are trying to maximize quality” — echoed later: “our goal is to maximize quality, not to minimize costs.” ## Uncertainty Notes - **Hedged attributions.** Gossip-style claims are flagged as secondhand: “Another thing you can do that I've heard stories of is you can actually hire a crowdsource army to vote for you…” - **Self-correction moments in live delivery:** “And I used to think that saturation was just them saying… And it often does mean that but it can mean…” — showing explicit revision of his own prior belief. - Transcription artifacts (“El Marina,” “ifal,” “Serge,” “Sweepbench”) appear to be ASR errors; speaker's actual referents (Arena, IFEval, Surge, SWE-Bench) are inferred from context and should be treated with that caveat. # Observation Record: "When Will The Benchmaxxing Plague End?" — Nick Heiner, Surge AI ## Speaker - Single speaker throughout: **Nick Heiner** (per video title metadata), presenting on behalf of Surge AI. ## Framing Claims - The AI industry loves hype cycles; benchmarks are cited at model launches, sometimes with "chart crime," and when reality fails expectations, allegations of **benchmaxxing** arise. - Defines benchmaxxing: labs "training too hard on benchmarks in a way that deviates from what people actually care about." - Poses four questions and answers them upfront: why benchmaxxing happens (**incentives**), why benchmarks misreflect real-world value (**poor methodologies**), whether it's intrinsic to all benchmarks (**no**), whether we'll ever know which models are best (**yes**). ## Why Bad Benchmarks Persist - Millions of dollars wagered on prediction markets on **LMArena** outcomes even as industry leaders "openly brag about gaming" it; "thought leaders like Worf" say it can be easily gamed. - Quotes Karpathy's observation (as reported by Heiner): teams are not getting better models overall but "better Elm Marina models... possibly something with a lot of nested list bullet points and emojis." - Argument: AI is for everyone, everyone needs a tool to judge models, benchmarks are that tool; those unable to assess benchmark quality default to popularity, creating an **incumbency/marketing-driven feedback effect** over real-world value. Heiner admits he himself lacks an opinion on a benchmark unless he studies it in detail. ## Benchmark Creation Anti-patterns ### Price - Cost math: a 1,000-task agentic coding benchmark at 60 hours/task with $500k/year engineers = **$15M** upfront, plus ~$5M/year replacement as tasks saturate. - Workarounds fail: heavy AI assistance "doesn't really work" — "you can't push the frontier forward from within the frontier"; cheap labor yields poor results. - Surge position: maximizes quality, not cost minimization; in 2026 "models are just beyond the point where you can make do with anything less than the best workers." ### Contamination - Contamination is "the default outcome unless you are very very good"; public Q&A on the internet gets memorized. - Claims an investigation found Opus clearly memorized substantial **SWE-bench Verified** content (vs. the rest of the repos) — can verbatim complete prompts and answers. - States Opus 4.8's model card reports its SWE score **without disclosing this contamination**, and the industry lacks disclosure norms, leaving consumers uninformed. ### Reward hacking - Models find lazy/creative ways to meet the letter, not spirit, of a benchmark; "gradient descent is like water flowing downhill." Verifiers must be adversarially robust. ### Lack of ambition / crude verifiers - **Automation Bench** example: hard-coded string-match verifiers (e.g., one acceptable phone number format, unspecified in the prompt). Result: Haiku and Fable both score 20% — Haiku from mistakes, Fable from format mismatches despite 80% correctness. A task not differentiating them is not useful. - In 2026, benchmarks should be the "lighthouse" for industry-remaking AI; string matching won't measure that. ### Taste / product sense - Modern benchmarks are "aspirational artifacts" expressing values; require product sense about desired AI behavior. - **IFEval** critique: built from arbitrary prompts no user has asked earnestly (e.g., "do not use commas," "use the letter T at most once"); requires an unproven generalization assumption; contains contradictory, unsolvable prompts (e.g., "repeat verbatim" + "translate into Hindi"; "exactly one bullet point" + "include a few bullet points"); sentence splitter misaligns with human segmentation; many prompts under-verified — a story prompt's verifier only checks the letter "I" appears at most once, so reward-hacked responses (Cyrillic І instead of ASCII I) get full scores. ### Operational ability / QC - **Apex** (RAG benchmark): file contents sometimes contradict the rubric, so correct agents get negative scores; data appears synthetically generated (placeholder values, nonexistent dates/places), risking **eval awareness** and taking models out of distribution. ## Lab-side Benchmaxxing - Human eval is the ground truth but expensive; benchmarks attempt faithful distillation of human preference — divergence point exists where benchmark scores climb while human eval stays flat or even **declines**, driven by marketing or organizational politics. - Example: prompt "what time is it?" with a "deranged" response — no human eval picks it, but LMArena ranks it top. - Claims (via "stories I've heard," i.e., secondhand) of hiring crowdsource armies to vote on LMArena, which "does no filtering of their workforce"; anonymization bypassed via model watermarks telling crowds who to vote for. - Non-representative or undisclosed eval conditions undermine community interpretation; cites a paper on LMArena dynamics where **Meta tested 27 models without disclosure**, distorting results. ## Prescriptions for Ending Benchmaxxing - Hold both benchmark makers and labs to a higher standard: - Start with **great human experts** informing tasks, success measures, inputs, tools. - Add **product/business sense** (e.g., medical benchmark needs regulatory/legal expertise, not just doctors). - **High-fidelity real-world input data** created by actual people; synthetic approaches possible but "very very hard to do reliably." - Working tools (bugs add noise unless bug-handling is the benchmark's purpose). - **Two-way verifier–prompt alignment**: verifiers check everything asked; everything asked is verified. - Thorough QC and a **private holdout set** to prevent contamination. - Reinterprets **saturation**: labs declaring 80% "saturated" may mean the remaining 20% of tasks are broken; while hill-climbing you can't tell which, and biased noise distorts model rankings. ## Surge AI's Hemingway Bench - Writing is "too rich and deep and nuanced and frankly human" for mechanical benchmarks; LLM-as-judge fails because "LLMs don't have good taste in writing" (frontier-from-within-frontier argument). - Solution: thousands of professional writers (technical writers, poets, journalists, editors) doing **blind model comparisons** for a leaderboard; expensive, but the goal is quality, not cost. ## Conclusion & Unresolved Tensions - Benchmaxxing is "the exploitation of benchmark misalignments between human preference"; both benchmark makers and those reporting on benchmarks can improve. - Closes with hiring pitch for Surge and a plug for his Substack. ## Qualifications to Preserve - LMArena vote-manipulation claim is explicitly secondhand ("I've heard stories of"). - Karpathy and "Worf" positions are reported by Heiner, not independently evidenced in the talk. - Contamination finding attributed to an investigation "we did," without methodology detail in the transcript. # Attribution Record — “When Will The Benchmaxxing Plague End?” ## Speaker Inventory | Speaker | Role | Notes | |---|---|---| | Nick Heiner | **Creator / Presenter** | Sole speaker throughout; gives the talk and frames all claims. Identified via title as affiliated with **Surge AI**. | - No guests, narrators, or other speakers appear in the transcript. ## Referenced Third-Party Claims (attributed within the talk) - **Andrej Karpathy** — quoted/paraphrased observation that models he considered best did not align with **LM Arena** (Elm Marina) rankings; quote includes "better Elm Marina models whatever that is possibly something with a lot of nested list bullet points and emojis." - **Yann LeCun ("Wor" / "Wör")** — cited as saying LM Arena "can be easily gamed." *(Name transcription uncertain in transcript.)* - **"Industry leaders"** — cited as "openly bragging about gaming" LM Arena (unnamed). - **A paper** — referenced regarding LM Arena dynamics and models overfitting; reported that **Meta** tested 27 models without disclosure. ## Benchmarks & Models Cited (subject of claims, not speakers) - **SWE-bench Verified** — claimed contamination; speaker states an investigation found Opus had memorized its contents; claims **Opus 4.8** model card reports a SWE score without disclosing contamination. - **LM Arena** — claims of gaming, vote manipulation via crowdsourced voters and watermarks, weak workforce filtering. - **Automation bench**, **IF-eval (ifal)**, **Apex** (RAG benchmark) — criticized for verifier/quality issues. - **Models named:** Opus / Opus 4.8, Haiku, Fable (Fable's existence is asserted by the speaker; not independently verified). ## Promotional Segments (flagged) - **Surge AI positioning** — multiple segments describe Surge as "not trying to minimize cost… maximize quality," paying "a lot of money for good workers." - **HemingwayBench pitch** — segment describing Surge's writing benchmark and professional-writer workforce. - **Hiring pitch** — explicit: *"obligatory pitch at Surge, we're hiring."* - **Substack promotion** — closing: "please follow my substack." ## Provenance Notes & Availability - **No curated metadata supplied** — speaker identity, date, and venue come only from the video title. - **Comment availability:** not indicated in supplied material. - Claims about labs' internal practices ("I've heard stories of") are explicitly **hearsay per the speaker**, not sourced documents. - The Meta/LM Arena paper is referenced without citation details. Let's get started. When will the benchmaxing plague end? In the tech industry, we love a hype cycle. And in AI, we really love a hype cycle. And the way we do that is when a model comes out, there's a big announcement, there's a lot of benchmark cited. Sometimes to keep things interesting, we do a little chart crime. And then people actually go and use it. And if the expectations aren't met by the reality, then we have allegations of benchmaxing. Benchmaxing, of course, being when labs are training too hard on benchmarks in a way that deviates from what people actually care about. So the existence of that term indicates that we have a sense that benchmarks don't always equal reality. And so in this talk we're going to figure out why does benchmaxing happen? Why are traditional benchmarks not always accurate reflections of real world value? Is this intrinsic to all benchmarks? And will we ever know which models are best? And the answers are incentives, poor methodologies, no and yes. All right, that was my talk. Thank you so much for coming. Um actually it looks like I have a few extra minutes so let's let's move on. I have a few extra slides we'll we'll go through. So we have a sense that benchmarks don't equal reality but the industry is dominated by a lot of popular but very bad benchmarks. So there's millions of dollars on prediction markets being wagered on Elm Marina outcomes even as we have industry leaders openly bragging about gaming Elm Marina and you have thought leaders like Wor saying it can be easily gamed. It's past time for the Elm Marina people to sit down and think about whether they're doing more harm than good. Andre Karpathy had a similar observation when he noticed that the models that he thought were best were not lining up with what Elmarina was ranking. He said unfortunately the teams are not getting better models overall but better Elm Marina models whatever that is possibly something with a lot of nested list bullet points and emojis. So why does this happen that sort of industry insiders are telling us that this benchmark is not useful but it still gets a lot of play. The problem is that AI is aimed at everyone in the world is is something everyone in the world can use. And so everyone needs some tool to figure out which models are best. And benchmarks are what we have for that. But if you can't if you don't have the ability to assess if a benchmark is good, what you do have is the ability to assess what's popular. And this creates this avalanche, this feedback effect where the conversation is very much driven by incumbency and marketing and less by real world value. and even myself, right? Like unless I actually look at a benchmark in a fair amount of detail, I don't have an opinion on it. So, it's a very challenging problem. So, what are the things that benchmarks do that lead to these problems? There are a handful of key antiatterns that we're going to go through. The first is price. Let's say you want to make an agentic coding benchmark, which these days is a very popular thing to want to do, and you want a thousand tasks in your benchmark. Each task takes 60 hours to make. Each software engineer in your workforce costs half a million a year. That's $15 million to make your benchmark. And if you think that over time about a third of those tasks are going to get washed away every year due to models getting better, that's $5 million to replace them. So that puts you out of budget for most projects. So then people turn to a variety of workarounds that have their own problems. One of which is trying to use a lot of AI assistance which ultimately does not really work. Like you can't push the frontier forward from within the frontier. You need to inject that external human expertise and it needs to be good expertise. If you try to use cheap labor, you're going to get what you pay for and the whole result is not going to be that useful. At Surge, one of our differentiators has long been that we are not trying to minimize cost. We are trying to maximize quality and part of that means paying a lot of money for good workers. We've always believed that but especially in 2026 models are just beyond the point where you can make do with anything less than the best workers. Contamination is often thought of as when labs are explicitly training on the test set and that does happen sometimes but really contamination is the default outcome unless you are very very good. So labs put a lot of effort into holding back this flood of data that's going to contaminate their models. But inevitably if you have public questions and answers on the internet that's going to get memorized to some extent. So SweetBench verified here's an example prompt. You can give opus the first part of the prompt and it will verbatim spit out the rest. It does that with the answers as well. And we actually did an investigation where we compared looking at the repos that Sweepbench verified was built out of. How much has Opus memorized the Sweepbench verified contents versus the rest of the repo? And we found very clear evidence that Opus had memorized a lot of Sweetbench. In the most recent model card, Opus 4.8 talks about its SWE score. It does not disclose this contamination. We as an industry aren't really in the habit of doing those disclosures. And so what that means is that as benchmarking consumers, we're just missing that information. Reward hacking is also a big problem. Reward hacking is basically when a model finds a lazy and creative way to meet the letter of the law, but not the spirit. You need to think about designing your rewards as a adversarial process against this maximally lazy agent. Gradient descent is basically like water flowing downhill looking for the path of least resistance. And so your verifiers need to be robust to that. Another key challenge is simply just not having the ambition to make a sophisticated enough benchmark. Automation bench tests that agents are able to make tool calls in an enterprise environment. The problem is that a lot of the verifiers are these hard-coded string matches. And so you'll see it for things like phone numbers where there are many different acceptable phone number formats. But this verifier just picks one and the prompt doesn't tell you which one it is. So the result of this is that Haiku and Fable both score 20% on this task. Haiku scores 20% because it makes a bunch of mistakes and Fable scores 20% because it gets it right 80% of the time but then just happens to pick different formats. So if the benchmark task is not differentiating between Haiku and Fable, it's not a useful task. And more broadly, in 2026, many of us in this room are looking towards AI that's about to remake entire industries. And benchmarks are ideally our lighthouse on the horizon to let us know when that's coming. And a simple hard-coded string match is just not going to do it to measure that sort of impact. Another important aspect of a good benchmark is taste. Perhaps it used to be the case that benchmarks were these dry academic, you know, questions. and answer sets. But nowadays, a benchmark is an artifact expressing what it's an aspirational artifact. It's an expression of values of what you want your AI to do and how you want it to behave. And so you need to have some product sense in this process, some sort of a sense of what you want the AI to do. And that sense is unfortunately missing from ifal if has been cited on many model cards. And the way it was constructed was taking a bunch of arbitrary prompts that no user has ever asked in earnest and mashing them up with a bunch of other prompts to create a prompt set. The problem is that because no user actually has asked do not use any commas in your response or use the letter T at most once. You have to believe for this to be useful, you have to believe that there's a generalization from this to actual things that users are going to ask. If eval just happens also to have a bunch of prompts that are fully unsolvable due to having contradictory instructions. So this one starts by saying repeat this response verbatim and it ends by saying translate this into Hindi. Obviously you can't do both of those at once. Here's one that says write a riddle that includes exactly one bullet point. Make sure to include a few bullet points. Again this is just fully impossible. It uses a sentence splitter that does not align with how humans would actually split the sentences. And a lot of the prompts are not fully verified. So this one says write a story. There's nothing in the verifier that checks that a story was written. It just checks that the asky character I is not used more than once, which means that all of these responses get a full score, including response D. The way it gets a full score is by reward hacking and using the cerrillic eye character instead of the asy eye character. If is totally fine with that another challenge is operational ability. Making a big benchmark requires a lot of QC work and plenty of organizations just don't make that investment. Apex is a rag benchmark where the agent is given files and then asked questions about them. And in some instances, what's in the file and then what's expected in the rubric don't line up. So an agent that does the thing that it's seeing in the ground truth is going to get a negative score. And a lot of the data in Apex is seemingly synthetically generated because it's full of obvious placeholder values or dates or places that don't exist. And so as a result, the model is more likely to develop eval awareness where it realizes that it's being tested which undermines the entire exercise. It also just takes you out of distribution from actual real world data to something that is obviously fake. So that's an overview of some of the key antiatterns that happen during benchmark creation. But benchmaxing is a two-way process and there are all sorts of fun things that labs can do to benchmax and that's what we're going to talk about next. So the the core value that we're all trying to get towards as human eval right AI exists to serve humans and so just having humans look at the responses and make ratings like that's what we care about. The problem is that human eval is very expensive. And so a lot of what benchmarks are doing is trying to get around that and you are trying to distill human preference into something more scalable and you're hoping you do that distillation in a way that's still sufficiently faithful to what human eval wants. But what this means is that inevitably there is a point where you can keep hill climbing on a benchmark and the human eval stays flat. And you can actually take it even further if you want where you keep hill climbing on a benchmark even as the human eval goes down. But if for whatever reason you think this is necessary for marketing or we have sort of organizational politics or incentives that are demanding this that's how it can end up happening. In this instance the prompt is what time is it? And the response is absolutely deranged. No human eval is ever going to choose this but El Marina puts it at the top of the leaderboard. So again, you have this divergence and if you're trying to benchmax, you just cannot care about that. Another thing you can do that I've heard stories of is you can actually hire a crowdsource army to vote for you in Elmarina since Elmarina basically does no filtering of their workforce. And you might say, well, we anonym, you know, Elmarina anonymizes. So how are they going to know who to vote for? That's actually quite simple. You have your model include a watermark that tells the crowd who to vote for. There's also all sorts of things you can do with running your evals in conditions that are like not fully representative of the applesto apples comparison you're trying to make and then not always being super transparent about those conditions in such a way that undermines the validity that the community is trying to interpret because they don't have that contextualizing information. This was a paper um again about Elmarina and talking about how some of the dynamics of how it's run lead to models overfitting on Elmarina. Um in this instance, the specific chart we're seeing is that Meta tested 27 models without disclosing that it was doing so. Um which you know distorts the results. So how are we going to end benchmaxing? We need to hold the benchmark industry and the labs to a higher standard. The first thing we need to do when making a good benchmark is start with great human experts. And those experts inform everything that is downstream from what types of tasks are we going to have the agent do? How is success measured? What are the input files that agents are given? What are the tools that they're given? But we also do need that product sense. So imagine you're making a medical benchmark. It's not enough to have doctors who can answer specific medical questions because if you're trying to test how ready are we for agents to be deployed into hospitals. You also need someone with the business sense to know what's the regulatory environment, what's the legal requirements because that is going to impact what types of tasks you're trying to have the AI solve. You need high fidelity input data which is best done by going out and getting it from the real world, having actual people create this data. Synthetic approaches are possible, but it is very very hard to do it reliably. The tools need to actually work. A lot of benchmarks have tools that are buggy in various ways. And unless you're intentionally making a benchmark about buggy tools, this just introduces noise. You need verifiers that are fully aligned with the prompts. And this is a two-way alignment. So the verifiers need to be verifying everything the prompt asks for. And everything the prompt asks for needs to be covered by the verifiers. And if you get either side of those two misaligned, then it's going to be unfair to models and you're introducing random noise. You need to thoroughly QC everything and you need to have a private hold out set so you don't get contaminated. And if you do all this right, then you'll avoid what often happens with benchmarks, which is when labs get to like 80% and say, "Okay, this is saturated." And I used to think that saturation was just them saying again we don't think training on this further is going to increase real world value. And it often does mean that but it can mean that because the lab is saying we realize 20% of these tasks are broken. But the problem is that as you're hill climbing you don't know what 20% are broken until you solve all the others. And so as a result you have a lot of noise. And if that 20% of broken tasks is randomly but in a biased way assigning the rewards, it's going to really distort the model relative ranking you're trying to get. So at Serge, we created a benchmark called Hemingway bench to measure writing. There have been a number of writing benchmarks that use various mechanical means to try to assess writing quality, but we believe that writing is just too rich and deep and nuanced and frankly human of an activity to measure with mechanical benchmarks and LM as a judge doesn't really work either because LLMs don't have good taste in writing. Again, this is sort of the you can't expand the frontier from within the frontier situation. So what we've done is we've just created a workforce of thousands of professional writers in various domains, technical writers, poets, journalists, editors, and we just have them do blind model comparisons and then we create this leaderboard and it is quite expensive, right? Human eval is very expensive. Getting the time of these professionals is quite expensive. But again, our goal is to maximize quality, not to minimize costs. So in conclusion, benchmaxing is the exploitation of benchmark misalignments between human preference, but we can do better and we can hold the industry to a higher standard. Both the people making the benchmarks like myself and the people who are reporting on the benchmarks. And if you'd like to be a part of that, of course, obligatory pitch at Serge, we're hiring for basically all aspects of that. Uh and if you'd like more spicy takes from me, uh please follow my substack. Thank you very much.