--- schema: 1 type: video title: 'Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i' description: >- Ali Khial took three of the best engineers at G2i, pointed them at popular coding benchmarks, and hit a wall of tasks that were either too ambiguous to grade or quietly broken. That experience is the spine of this talk: a benchmark starts as a spec, solutions get verified and graded, and the results rank models, but only if the harness is actually creating a fair test rather than an unfair one. He shows real examples where an instruction is so vague that a correct patch gets rejected, or a test checks something as arbitrary as how a variable is named, and notes that a meaningful share of tasks he examined had genuinely good answers marked wrong. The danger is that models are increasingly good at gaming exactly this, hunting down the test and satisfying it rather than solving the problem, which opens a quality gap that public leaderboards hide. Khial lays out the principles he now uses for benchmarks worth trusting: be precise where precision matters and loose where it does not, keep a private held out set so nothing leaks from public GitHub repos, and hold the whole thing to production grade. His point is not that benchmarks are useless but that the ones we lean on are not there yet, and building better ones is the work. Speaker info: - https://www.linkedin.com/in/ali-khial/ Timestamps: 0:00 - The good, the bad, and the ugly 1:27 - Testing with our best engineers 2:30 - A benchmark as a spec 3:37 - When instructions are too ambiguous 4:44 - Tests that check the wrong thing 6:12 - Good answers marked wrong 7:03 - Models learning to game the test 8:08 - The quality gap leaderboards hide 9:03 - Precise where it matters 10:47 - Keeping a private held out set 11:13 - Principles for benchmarks worth trusting canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/benchmarks-the-good-the-bad-and-the-ugly-ali-khial-e871c4f0b32c markdown: >- https://liquidgold.cc/channel/aidotengineer/video/benchmarks-the-good-the-bad-and-the-ugly-ali-khial-e871c4f0b32c.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T06:29:33.380Z' language: null children: [] videoId: jWq-aZIU0kM durationSeconds: 769 publishedAt: '2026-07-31T16:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 90 endSeconds: 101 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=90' heldReason: rights - ordinal: 1 text: null startSeconds: 101 endSeconds: 150 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=101' heldReason: rights - ordinal: 2 text: null startSeconds: 162 endSeconds: 222 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=162' heldReason: rights - ordinal: 3 text: null startSeconds: 232 endSeconds: 289 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=232' heldReason: rights - ordinal: 4 text: null startSeconds: 232 endSeconds: 289 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=232' heldReason: rights - ordinal: 5 text: null startSeconds: 299 endSeconds: 376 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=299' heldReason: rights - ordinal: 6 text: null startSeconds: 387 endSeconds: 434 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=387' heldReason: rights - ordinal: 7 text: null startSeconds: 387 endSeconds: 434 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=387' heldReason: rights - ordinal: 8 text: null startSeconds: 490 endSeconds: 501 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=490' heldReason: rights - ordinal: 9 text: null startSeconds: 544 endSeconds: 556 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=544' heldReason: rights - ordinal: 10 text: null startSeconds: 556 endSeconds: 651 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=556' heldReason: rights - ordinal: 11 text: null startSeconds: 556 endSeconds: 651 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=556' heldReason: rights - ordinal: 12 text: null startSeconds: 664 endSeconds: 676 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=664' heldReason: rights - ordinal: 13 text: null startSeconds: 676 endSeconds: 686 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=676' heldReason: rights - ordinal: 14 text: null startSeconds: 686 endSeconds: 769 watch: 'https://liquidgold.cc/watch/jWq-aZIU0kM?t=686' heldReason: rights --- # Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i Ali Khial took three of the best engineers at G2i, pointed them at popular coding benchmarks, and hit a wall of tasks that were either too ambiguous to grade or quietly broken. That experience is the spine of this talk: a benchmark starts as a spec, solutions get verified and graded, and the results rank models, but only if the harness is actually creating a fair test rather than an unfair one. He shows real examples where an instruction is so vague that a correct patch gets rejected, or a test checks something as arbitrary as how a variable is named, and notes that a meaningful share of tasks he examined had genuinely good answers marked wrong. The danger is that models are increasingly good at gaming exactly this, hunting down the test and satisfying it rather than solving the problem, which opens a quality gap that public leaderboards hide. Khial lays out the principles he now uses for benchmarks worth trusting: be precise where precision matters and loose where it does not, keep a private held out set so nothing leaks from public GitHub repos, and hold the whole thing to production grade. His point is not that benchmarks are useless but that the ones we lean on are not there yet, and building better ones is the work. Speaker info: - https://www.linkedin.com/in/ali-khial/ Timestamps: 0:00 - The good, the bad, and the ugly 1:27 - Testing with our best engineers 2:30 - A benchmark as a spec 3:37 - When instructions are too ambiguous 4:44 - Tests that check the wrong thing 6:12 - Good answers marked wrong 7:03 - Models learning to game the test 8:08 - The quality gap leaderboards hide 9:03 - Precise where it matters 10:47 - Keeping a private held out set 11:13 - Principles for benchmarks worth trusting Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/benchmarks-the-good-the-bad-and-the-ugly-ali-khial-e871c4f0b32c · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:12 to 1:41) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (1:41 to 3:42) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (3:42 to 4:49) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (4:49 to 6:27) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (6:27 to 8:10) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (8:10 to 8:21) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (8:21 to 10:51) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (10:51 to 11:26) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 8: Document excerpt (11:26 to 12:49) > Held: transcript rights do not permit this chunk's text.