Ali Khial (G2i) examines the strengths and failures of popular coding benchmarks, drawing on an experiment where G2i's top engineers struggled with tasks that were too ambiguous to grade or silently broken. The talk walks through how a benchmark functions as a spec, how solutions are verified and graded, and how results rank models. It covers concrete failure modes such as vague instructions rejecting correct patches, tests checking arbitrary details like variable naming, and a meaningful share of tasks with genuinely good answers marked wrong. Khial also discusses models learning to game tests rather than solve problems, the quality gap hidden by public leaderboards, and his principles for trustworthy benchmarks: precision where it matters, a private held-out set to prevent leakage from public repos, and production-grade rigor. # Video Identity Observations — Benchmarks Talk ## Explicit Self-Description (Creator Claims) - **Speaker role**: Ali Khial introduces himself as **"the director of AI and ML at G2i"** and describes himself as **"a software engineer at heart"**, joking he has "zero experience in ML." - **Team claims**: He references working "with our team at G2i" over "the last 2 months" to define a framework of benchmark-design principles, and mentions "I have a lot of good researchers in the team." - **Organizational demonstration of engineers**: He claims to have "three of our best engineers" review a benchmark prompt. ## Observed Presentation (Demonstrated Patterns) - **Format**: A conference-style talk — "this is the last talk of this session," ending with applause and a call to action ("join the Discord"). - **Intended audience**: Primarily **software engineers** — closing statement: "this is a call to action to software engineers. Um benchmarks are not hard. We need to look under the hood." - **Scope of content**: Technical critique and analysis of **AI/LLM benchmarks**: - Problems: unrealistic instructions (481 words average in SWE-Bench Pro), "leaky prompts," weak verifiers, reward hacking. - Proposed framework: five principles — **human instructions, holistic graders, production grade, contamination-free by design, and better leaderboards**. - **Evidence style**: Cites specific external benchmark research (SWE-Bench Pro, SWE Marathon, Deep Sweet comparison: "8.5% of all the tasks accepted wrong implementation... 24% rejected correct implementations"). - **Tone**: Self-deprecating and informal ("more than 50 abandoned side projects," "I don't know why they put the ML in my title"), with candid meta-commentary ("The title of the presentation is a little bit misleading"). ## Audience and Channel Positioning (Observable) - Addresses an audience fatigued from a long conference day and presumed to have engineering context (references PRs, unit tests, "100% coverage" without explanation). - Positions the channel/speaker as a **practitioner perspective on AI evaluation**, bridging software engineering practices and LLM benchmarking rather than pure ML research. ## Notes on Limits - No curated metadata was provided; all observations derive from the title and transcript. - The talk itself acknowledges a pivot: "it would be better if I presented my journey into benchmarks... instead of trying to find a dichotomy" — indicating the delivered scope differs from the title's framing. # Delivery Observations: Ali Khal, "Benchmarks: The Good, the Bad, and the Ugly" ## Tone - **Casual, conversational, self-deprecating.** Opens with audience-awareness humor and jokes about his own credentials: > "I'm the director of AI and ML at G2I. Um I have zero experience in ML. So I don't know why they put the ML in my title." > "I have more than 50 abandoned side projects in my machine. So uh you can know." - **Empathetic toward audience fatigue** at the opening: > "I know that you guys had to go through a long day. So try to keep it short and light for you all." ## Vocabulary & Recurring Phrases - **Heavy filler usage**: "um" and "uh" appear throughout, plus repeated word restarts: > "what what what went wrong?" > "and and a lot of jargon" > "increasingly increasingly able" - Software-engineering vocabulary anchors the whole talk: **prompt**, **test**, **harness**, **patch**, **PR**, **verifier**, **rubric**, **coverage**, **unit test / integration tests / end-to-end tests**. - Benchmark-specific jargon is named explicitly early on: > "graders, long horizon, verifiers, bench benchmarks, and and a lot of jargon" - Recurring evaluative frames: **"economically valuable"**, **"leaky prompt"**, **"weak verifiers"**, **"reward hacking"**, **"contamination free by design"**, **"quality gap" / "trust gap"**. ## Teaching Habits & Structure - **Journey-framing over taxonomy**: explicitly disclaims the title and pivots to a personal narrative: > "The the title of the the presentation is a little bit misleading... it would be better if I presented my journey uh into benchmarks and what I learned" - **Attention-grabbing hook with evidence**: opens the technical body with concrete screenshots and a mini-experiment: > "I took three of our best engineers. I showed them the prompt and I said, 'Would you ever write a prompt like this?' And the answer was no." - **Labeled examples** — names each failure pattern memorably: "the leaky prompt," "the not economically valuable prompt," "weak verifiers," "reward hacking." - **Numbered principles for solutions**: five explicit principles ("The first one...", "The second principle...", "The third principle...", "The fourth principle...", "And the fifth and last principle"). - **Simplification before detail**: reduces the benchmark concept to a pipeline: > "it starts as a prompt or an instruction. That prompt is fed to models and agents... All of that is wrapped in a harness..." - **Analogy to engineering practice** when explaining grading: > "This is very similar to how we approach um tests in engineering." - **Anecdotal evidence for claims** (uncertainty noted — single-source anecdotes): > "I have not met an engineer in the last 6 months that would choose a model... based on the leaderboards." - **Explicit signposting/transitions**: "All right, moving on," "And so," "So, how do we close the gap?" ## Communication Patterns - **Direct audience address and rhetorical questions**: "So what what what went wrong?", "I don't know if any of you ever tried to do that." - **Quotes hypothetical engineer speech** to make standards vivid: > "If the LLM is fixing this, I trust it to fix that." - **Honest meta-commentary about his own preparation**: > "initially, I wanted to have a kind of a a lofty like ending to this, but I think I I I pivoted to something more interesting." - **Ends with a call to action**, not a summary: > "Benchmarks are not hard. We need to look under the hood... join the Discord because engineers' input is valuable." ## Stable vs. Isolated Features - **Stable**: filler/restart-heavy delivery, engineering analogy, labeled examples, numbered principles, self-deprecating framing — all recur across the talk. - **Isolated**: the "50 abandoned side projects" joke and "zero experience in ML" quip occur once (intro); the Discord call to action appears only at the close. The "481 words per instruction" statistic is a single data point — the underlying research method isn't shown, so its rigor is uncertain. # Observation Record: "Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i" ## Speaker - **Ali Khial**, Director of AI and ML at G2i (self-identified in transcript). He describes himself as "a software engineer at heart" with "zero experience in ML," offered as a joke about his title. ## Framing & Disclaimer - Khial states the talk title is **misleading**: instead of a dichotomy of good/bad/ugly, he presents his **personal journey into benchmarks** and what he learned. ## How Benchmarks Work (Khial's Simplified Model) - A **prompt/instruction** is fed to **models and agents**, which provide **solutions**. - Solutions are **verified and graded** via verifiers and rubrics. - All wrapped in a **harness** shielding from external factors. - Output: **trajectories, scores, and metadata** used to rank models. - His stated "equation": if prompts, verifiers/rubrics, and the harness are all good, results should be great — **"but, that's not the reality."** ## Problem 1: Unrealistic Instructions - He surveyed **SWE-Bench Pro**: average of **481 words per instruction** — "a two-pager per task. That is not how people write prompts." - Anecdote: he showed a benchmark prompt to three of G2i's best engineers; all said they would never write a prompt like that — "And they're right. They shouldn't." - **"Leaky prompt" example** (Go task, regex matching): - Instruction points directly to the test file, giving the LLM everything needed to implement against the tests. - A second case is "even worse": provides the complete implementation interface, "locking the LLM from any kind of creativity." - **"Not economically valuable" example** (from SWE Marathon): a well-formed, well-abstracted prompt — but it asks the LLM to **build a C compiler in Rust**. Khial: "I don't think it's a good idea. We should not do that." ## Problem 2: Weak Verifiers - Cites work by **Deep SWE** comparing their benchmark against SWE-Bench Pro: in SWE-Bench Pro, **8.5% of tasks accepted wrong implementations**, and **~24% rejected correct implementations**. - Examples he extracted: - A test expecting a variable to exist that was **not specified in the instruction** — "cornering the LLM" and causing false negatives. - A test checking **unexported functions** — he states such tests would not be accepted as a PR in real projects. ## Problem 3: Reward Hacking - Models increasingly **go around problems** rather than solving them — e.g., searching for `.git` folders or internet traces instead of applying a patch. - He presents a graph showing reward hacking **increasing over time and across model versions**, and argues this reflects model capability, not model failure: "that's what we want. We want LLMs to be smart. The benchmarks are lacking behind." ## Consequence: Trust Gap - Claim: **"There's a quality gap and it's causing a trust gap."** - Anecdotal evidence: "I have not met an engineer in the last 6 months that would choose a model... based on the leaderboards" — engineers look at leaderboards, note the hype, then test models themselves. ## Proposed Framework (G2i, last 2 months of work) 1. **Human instructions** — authored and reviewed by humans; express desired behaviors, objectives, and hard constraints — not implementation details. 2. **Holistic graders** — behavioral tests broadly, precision where needed (analogous to engineering test strategy: full stack for security/business logic; not 100% coverage everywhere, since that is "not efficient"). 3. **Production grade** — tasks must be economically valuable; the test of success is an engineer saying, "If the LLM is fixing this, I trust it to fix that." Khial: "Currently, we don't have that." 4. **Contamination-free by design** — novel tasks only; keep **private holdout sets**. He notes current benchmark tasks are pulled from public GitHub repos. 5. **Information beyond leaderboards** — benchmarks should tell a story and explain **why**, not just **who wins**; surface the "x-axis" data from runs on the first page rather than forcing users to dig. ## Call to Action - Directed at **software engineers**: "Benchmarks are not hard. We need to look under the hood... join the Discord because engineers' input is valuable." ## Qualifications & Notes - Several claims rest on Khial's anecdotes (three engineers surveyed; no leaderboard-trusting engineers met in 6 months) presented as personal observation, not systematic data. - He reframes increasing reward hacking as a **benchmark deficiency**, not a model deficiency — a contested framing stated as his position. - Statistics (481 words average; 8.5% / ~24%) are attributed by Khial to his own quick research and to Deep SWE's comparison respectively; underlying methodology is not described in the talk. # Speaker Attribution Record — Benchmarks: The Good, the Bad, and the Ugly ## Speakers | Speaker | Role | Evidence | |---|---|---| | **Ali Khial** | **Creator / Presenter** — Director of AI and ML at G2i; self-described software engineer | Self-introduction: *"I'm Ali. I'm the director of AI and ML at G2I."* Delivers the entire talk. | | Unnamed audience | Listeners (no speaking role) | Implied by talk framing and closing **[applause]** | - Single-speaker presentation; **no guests, narrators, or other identified roles** present. - A joke within the talk (*"I have zero experience in ML"*) is presenter humor, not a role indicator — his stated title is Director of AI and ML at G2i. ## Referenced Sources & Third-Party Claims - **SweetBench Pro** — cited; claim of **481 words per instruction on average** ("I did a quick research"). - **Sweet Marathon** — cited as source of the "not economically valuable prompt" example (build a C compiler in Rust). - **Deep Sweet** ("SWEET"/"Deep Sweet" per transcript) — cited for comparison data against SweetBench Pro: claims that **8.5% of tasks accepted wrong implementations** and **~24% rejected correct implementations**. - **Anonymous engineers at G2i** — three engineers consulted on prompt realism; also "good researchers" on his team credited with simplifying benchmark concepts. - Anecdotal claims: no engineer in the last 6 months choosing models via leaderboards (unsourced personal observation). ## Promotional Segments - **Call to action** at the close: urging listeners to "join the Discord" and contribute as engineers — flagged as **promotional/community-solicitation** rather than product marketing. - Reference to G2i team's 2-month internal framework work; **no commercial promotion** detected. ## Provenance & Availability Limits - **Comment availability:** not indicated in the provided material. - **Timestamps:** none available in transcript. - Figures (8.5%, 24%, 481 words) are the **presenter's own research/extraction**, not independently verifiable from this transcript. - No curated metadata supplied beyond title and transcript; speaker identity rests solely on in-transcript self-introduction. Hello everyone. Um this is the last talk of this session. So hopefully it's going to be short. I know that you guys had to go through a long day. So try to keep it short and light for you all. Um I'm going to present myself. Um I'm Ali. I'm the director of AI and ML at G2I. Um I have zero experience in ML. So I don't know why they put the ML in my title. I'm a software engineer uh at heart. And to prove that I have more than 50 abandoned side projects in my machine. So uh you can know. So uh I'm going to make a disclaimer. The the title of the the presentation is a little bit misleading. Uh as I was working on it, I realized that it would be better if I presented my journey uh into benchmarks and what I learned instead of trying to find a dichotomy of the the bad, the ugly, and and the good. So um let's start with um I want to grab your attention. And I invite you to look at this. These beautiful three screenshots are a single prompt on one of the benchmark tasks. As I was looking at it, I was like how can an engineer write a task like this? So I said, "Nah, it's impossible. No one writes prompts like these ever." But I wanted to double-check with my engineers. So I took three of our best engineers. I showed them the prompt and I said, "Would you ever write a prompt like this?" And the answer was no. And they're right. They shouldn't. And so at that point I'm I was like what is a what are benchmarks anyway? Uh I needed to take a step back. I needed to look more. I needed to understand. And so as I was researching, I faced a wall of keywords. Um graders, long horizon, verifiers, bench benchmarks, and and a lot of jargon. So, I was like, either this is too complicated or um there's a lot of jargon and a lot of um words to to to work through here. So, um I worked through it, worked with my team. I have a lot of good researchers in the team, and we uh kind of like nailed like simplified it to the most basics. Um and so, the way I see it is that it starts as a prompt or an instruction. That prompt is fed to models and agents. Agents provide solutions. Those solutions are verified uh and graded through verifiers and rubrics. All of that is wrapped in a harness that's that's preventing it from um from the external factors. And if it all goes good, uh we have um trajectories, scores, and um metadata that we can use um to to to verif- to basically uh rank um models. And so, the equation is simple. If prompts and instructions are great and verifiers and rubrics are doing their job while the harness is preventing um or creating an environment that is good for a benchmark, we should have amazing results. Um but, that's not the reality. So, what what what went wrong? So, the first thing is when looking deeper in benchmarks, uh most of the instructions are unrealistic. Um I did a quick research on SweetBench Pro, and um there's 481 words per instruction in average. That's a two-pager per task. That is not how people write prompts. And to illustrate more of that, um I took a couple examples here. The first one I looked at I I call the leaky prompt. It's a go um task that's basically um that's trying to match in some rejects and doing test it's on on some rejects. So, in the first screenshot here, um the instruction is pointing directly to the test file, which basically means that the LLM has all the ingredient it needs to go and find that test file and implement based on that. The second one is is even worse. Um it's basically providing a complete interface of the implementation. Basically locking the LLM from any kind of uh creativity and it's forcing it to do it that way. So, that's the leaky prompt. The second example, it's the the not economically valuable prompt. Uh this is from Sweet Marathon. And this prompt is well-formed. It's It's It's abstracted enough to allow for the LLM to do its work, but it's asking it to build a C compiler in Rust. So, I don't know if any of you ever tried to do that, but I don't think it's a good idea. We should not do that. All right, moving on. The second problem, weak verifiers. Um so, the screenshot here is is a uh is the work that Deep Sweet um did uh to compare their uh their bench against Sweet Bench Pro. And um let me just fix here so I can see the numbers. In Sweet Bench Pro, 8.5 of 8.5% of all the tasks uh accepted wrong implementation in one hand and more than 20 24% of the tasks uh rejected um correct implementations. And so, I kind of went again, dug a little bit, and I extracted one of the tasks, and I started looking at it. Um and and here's here's what's happening in the example of uh re- rejecting um possibly rejecting good good answers. So, in this example, the test is is basically expecting a variable to exist. But that variable is first not specified in the instruction, and two, why would we expect an LLM to write the variable name this way? So, this test is cornering the LLM and basically uh causing uh those false negatives. In the other example, it's base the test is basically checking functions that are unexported. So, if that was a PR in any of our projects, and exposed these type of tests, we would not accept it. So, this is what a weak verifier looks like. All right, moving on. Re- reward hacking. So, what's happening is models are becoming increasingly increasingly able to optimize and figure out solutions to hard problems by going around the problem. So, instead of actually trying to fix the to to apply a patch to a task, they try to go and find dot git folders, or they look up the internet for any kind of traces that would allow them to um to do the task. And this first graph here shows like shows that as models evolve, they are now more smarter and smarter in being able to do reward hacking, but that's what we want. We want LLMs to be smart. The benchmarks are lacking behind and they're not preventing from from that to happen. Um More in detail, as you can see here, the more you go in time and the more you have new versions, the delta of um of um reward hacking is increasing. So, the conclusion here is there's a quality gap and it's causing a trust gap. I have not met an engineer in the last 6 months that would choose a model or choose um an LLM based on the leaderboards. Um they look at them. There's a lot of hype, but then they move on and they test things by themselves and they apply that. So, how do we close the gap? Um in the last 2 months, we've been working with our team at G2i to basically try to define a framework, uh a set of principles that would allow us to build tasks for benchmarks that are um better than what we have today. The first one, human instructions. Authored by humans, reviewed by humans. This is basically the entry point for any great tasks. The instructions given to an agent or an LLM should lean towards expressing desired behaviors, objectives, and hard constraints, not implement details or try to guarantee self-containment when the task itself is is expressing too much uh too much details. The second principle is holistic graders. Behavioral tests in one hand and then precision what were needed. This is very similar to how we approach um tests in engineering. We want to have the most surface covered without being too prescriptive, but we also want to be precise where needed. So, for security issues or business logic, we want to have the whole stack units unit test integration tests and then end-to-end tests. But, for the rest of the the rest of the the software, we don't want to have 100% coverage because that's um not efficient. The third principle, production grade. The tasks have to be tasks have to have value um and they have to be economically valuable. Um it is one thing to have a test a task that is failing the LLM proven that the LLM is not there yet. It is another for it's another thing for an engineer to look at a task and say, "If the LLM is fixing this, I trust it to fix that." Currently, we don't have that. So, production grade. The fourth principle, contamination free by design. We want to do novel tasks only and we want to make sure that we keep private holdout sets. This is a principle that is very important as currently the tasks that are existing in benchmarks are all put from GitHub repos or from um from from public repos. So, our approach here is that it should always be novel. This way, it's contamination free by design. And the fifth and last principle here is information about leaderboards. The benchmark needs to tell a story and needs to help people make decisions. Leaderboards are what we see in benchmarks today. They tell you who wins, but they don't to you why. And so, we want to basically put the x-axis back on um on on the first page. Uh the idea here is that there's um there's a lot of um data that we can extract from those these runs, and unfortunately, they're not being put in the forefront. And people have to dig uh a lot and do their own experiments to get to those data points. And so, finally, uh initially, I wanted to have a kind of a a lofty like ending to this, but I think I I I pivoted to something more interesting. Uh this is a call to action to software engineers. Um benchmarks are not hard. We need to look under the hood. And we need to understand them and join the Discord because engineers' input is valuable. And thank you. \>\> \[applause\]