--- schema: 1 type: video title: 'DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve' description: >- DeepSWE is 113 software engineering tasks written from scratch, not scraped from pull requests, so a model cannot have seen them in training. Each one is a long horizon problem drawn from a real open source repository, authored by engineers who actually maintain that code, with isolated environments and program based verifiers that check observable behavior rather than trusting the model's own account. James Shi's point is that once you remove the contamination the leaderboard stops clustering: strong models pull far ahead and others, Gemini 3.1 Pro among them, fall toward the bottom. The more revealing signal is in how models fail. Some quietly expand a task beyond what was asked, a failure mode DeepSWE scores in its own right, and Claude models did this a good fraction of the time while GPT models did it less often. Stronger models also tend not to verify their own work, and there is a real gap between the ones that test what they wrote and the ones that assume it is correct. Since reward hacking is a constant temptation, the verifiers are built to be gamed as little as possible, keeping the score anchored to the objective rather than to a convincing looking rollout. Speaker info: - https://x.com/shiqyy - https://www.linkedin.com/in/jamesshi117/ - https://deepswe.datacurve.ai Timestamps: 0:00 - Introduction: the DeepSWE benchmark 1:03 - 113 original, contamination-resistant tasks 2:08 - What makes a good benchmark 3:51 - The leaderboard and model spread 5:18 - Failure mode: over-scoping the task 7:16 - Do models verify their own work? 8:45 - Tasks authored by core contributors 10:15 - Writing realistic, high level prompts 11:45 - Program based verifiers and observable behavior 13:43 - Limitations and future work 15:25 - Reward hacking and keeping it cheating proof --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/Yk87oUPVaxU-deepswe-a-contamination-resistant-coding-benchmark-james-shi-datacurve Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/Yk87oUPVaxU' markdown: 'https://liquidgold.cc/watch/Yk87oUPVaxU.md' llms: 'https://liquidgold.cc/watch/Yk87oUPVaxU/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T06:30:08.151Z' language: null children: [] videoId: Yk87oUPVaxU durationSeconds: 1054 publishedAt: '2026-07-26T18:10:56.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 23 endSeconds: 70 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=23' heldReason: rights - ordinal: 1 text: null startSeconds: 70 endSeconds: 81 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=70' heldReason: rights - ordinal: 2 text: null startSeconds: 81 endSeconds: 138 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=81' heldReason: rights - ordinal: 3 text: null startSeconds: 81 endSeconds: 138 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=81' heldReason: rights - ordinal: 4 text: null startSeconds: 150 endSeconds: 240 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=150' heldReason: rights - ordinal: 5 text: null startSeconds: 150 endSeconds: 240 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=150' heldReason: rights - ordinal: 6 text: null startSeconds: 150 endSeconds: 240 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=150' heldReason: rights - ordinal: 7 text: null startSeconds: 251 endSeconds: 318 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=251' heldReason: rights - ordinal: 8 text: null startSeconds: 251 endSeconds: 318 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=251' heldReason: rights - ordinal: 9 text: null startSeconds: 328 endSeconds: 441 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=328' heldReason: rights - ordinal: 10 text: null startSeconds: 328 endSeconds: 441 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=328' heldReason: rights - ordinal: 11 text: null startSeconds: 441 endSeconds: 452 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=441' heldReason: rights - ordinal: 12 text: null startSeconds: 530 endSeconds: 540 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=530' heldReason: rights - ordinal: 13 text: null startSeconds: 540 endSeconds: 619 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=540' heldReason: rights - ordinal: 14 text: null startSeconds: 630 endSeconds: 709 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=630' heldReason: rights - ordinal: 15 text: null startSeconds: 709 endSeconds: 721 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=709' heldReason: rights - ordinal: 16 text: null startSeconds: 721 endSeconds: 832 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=721' heldReason: rights - ordinal: 17 text: null startSeconds: 832 endSeconds: 844 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=832' heldReason: rights - ordinal: 18 text: null startSeconds: 844 endSeconds: 935 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=844' heldReason: rights - ordinal: 19 text: null startSeconds: 935 endSeconds: 946 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=935' heldReason: rights - ordinal: 20 text: null startSeconds: 946 endSeconds: 1054 watch: 'https://liquidgold.cc/watch/Yk87oUPVaxU?t=946' heldReason: rights --- # DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve DeepSWE is 113 software engineering tasks written from scratch, not scraped from pull requests, so a model cannot have seen them in training. Each one is a long horizon problem drawn from a real open source repository, authored by engineers who actually maintain that code, with isolated environments and program based verifiers that check observable behavior rather than trusting the model's own account. James Shi's point is that once you remove the contamination the leaderboard stops clustering: strong models pull far ahead and others, Gemini 3.1 Pro among them, fall toward the bottom. The more revealing signal is in how models fail. Some quietly expand a task beyond what was asked, a failure mode DeepSWE scores in its own right, and Claude models did this a good fraction of the time while GPT models did it less often. Stronger models also tend not to verify their own work, and there is a real gap between the ones that test what they wrote and the ones that assume it is correct. Since reward hacking is a constant temptation, the verifiers are built to be gamed as little as possible, keeping the score anchored to the objective rather than to a convincing looking rollout. Speaker info: - https://x.com/shiqyy - https://www.linkedin.com/in/jamesshi117/ - https://deepswe.datacurve.ai Timestamps: 0:00 - Introduction: the DeepSWE benchmark 1:03 - 113 original, contamination-resistant tasks 2:08 - What makes a good benchmark 3:51 - The leaderboard and model spread 5:18 - Failure mode: over-scoping the task 7:16 - Do models verify their own work? 8:45 - Tasks authored by core contributors 10:15 - Writing realistic, high level prompts 11:45 - Program based verifiers and observable behavior 13:43 - Limitations and future work 15:25 - Reward hacking and keeping it cheating proof --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/Yk87oUPVaxU-deepswe-a-contamination-resistant-coding-benchmark-james-shi-datacurve Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/Yk87oUPVaxU · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:12 to 1:10) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (1:10 to 2:18) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (2:18 to 4:00) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (4:00 to 5:18) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (5:18 to 7:21) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (7:21 to 8:50) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (8:50 to 10:19) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (10:19 to 11:49) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 8: Document excerpt (11:49 to 13:52) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 9: Document excerpt (13:52 to 15:35) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 10: Document excerpt (15:35 to 17:34) > Held: transcript rights do not permit this chunk's text.