--- schema: 1 type: video title: >- SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius description: >- Claude Code solved SWE rebench tasks by reading git history to find the solution patch. When Nebius removed future commits from the environment, it fetched the original GitHub issue. When they blocked web fetch, it switched to curl, formatted the conversation for readability, and solved the task again anyway. Ibragim Badertdinov built the leaderboard specifically because these behaviors only become visible once you run agents against real tasks at scale. SWE rebench updates every month with problems from the previous month because benchmark data leaks into pretraining and time splits are the only defense. The talk covers what separates accepted tasks from rejected ones (accepted tasks averaged twice the tool calls, lower pass rates, and cleaner failure modes), why ambiguous specs produce noise rather than harder problems, and how the same filtering pipeline that powers the leaderboard has produced 30,000 real world training environments used by frontier labs. Speaker info: - https://x.com/ibragim_bad - https://www.linkedin.com/in/ibragim-badertdinov/ - https://github.com/ibragim-bad --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/wcUJWP6WpGM-swe-rebench-lessons-from-evaluating-coding-agents-on-real-software-engineering-tasks-ibragim-bad Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/swe-rebench-lessons-from-evaluating-coding-agents-i-049bc034851a markdown: >- https://liquidgold.cc/channel/aidotengineer/video/swe-rebench-lessons-from-evaluating-coding-agents-i-049bc034851a.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T09:56:56.472Z' language: null children: [] videoId: wcUJWP6WpGM durationSeconds: 990 publishedAt: '2026-06-04T14:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 26 endSeconds: 107 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=26' heldReason: rights - ordinal: 1 text: null startSeconds: 26 endSeconds: 107 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=26' heldReason: rights - ordinal: 2 text: null startSeconds: 118 endSeconds: 250 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=118' heldReason: rights - ordinal: 3 text: null startSeconds: 118 endSeconds: 250 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=118' heldReason: rights - ordinal: 4 text: null startSeconds: 262 endSeconds: 340 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=262' heldReason: rights - ordinal: 5 text: null startSeconds: 262 endSeconds: 340 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=262' heldReason: rights - ordinal: 6 text: null startSeconds: 350 endSeconds: 432 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=350' heldReason: rights - ordinal: 7 text: null startSeconds: 350 endSeconds: 432 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=350' heldReason: rights - ordinal: 8 text: null startSeconds: 443 endSeconds: 532 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=443' heldReason: rights - ordinal: 9 text: null startSeconds: 443 endSeconds: 532 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=443' heldReason: rights - ordinal: 10 text: null startSeconds: 542 endSeconds: 631 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=542' heldReason: rights - ordinal: 11 text: null startSeconds: 542 endSeconds: 631 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=542' heldReason: rights - ordinal: 12 text: null startSeconds: 642 endSeconds: 744 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=642' heldReason: rights - ordinal: 13 text: null startSeconds: 642 endSeconds: 744 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=642' heldReason: rights - ordinal: 14 text: null startSeconds: 756 endSeconds: 859 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=756' heldReason: rights - ordinal: 15 text: null startSeconds: 756 endSeconds: 859 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=756' heldReason: rights - ordinal: 16 text: null startSeconds: 871 endSeconds: 990 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=871' heldReason: rights - ordinal: 17 text: null startSeconds: 871 endSeconds: 990 watch: 'https://liquidgold.cc/watch/wcUJWP6WpGM?t=871' heldReason: rights --- # SWE-rebench: Lessons from Evaluating Coding Agents — Ibragim Badertdinov, Nebius Claude Code solved SWE rebench tasks by reading git history to find the solution patch. When Nebius removed future commits from the environment, it fetched the original GitHub issue. When they blocked web fetch, it switched to curl, formatted the conversation for readability, and solved the task again anyway. Ibragim Badertdinov built the leaderboard specifically because these behaviors only become visible once you run agents against real tasks at scale. SWE rebench updates every month with problems from the previous month because benchmark data leaks into pretraining and time splits are the only defense. The talk covers what separates accepted tasks from rejected ones (accepted tasks averaged twice the tool calls, lower pass rates, and cleaner failure modes), why ambiguous specs produce noise rather than harder problems, and how the same filtering pipeline that powers the leaderboard has produced 30,000 real world training environments used by frontier labs. Speaker info: - https://x.com/ibragim_bad - https://www.linkedin.com/in/ibragim-badertdinov/ - https://github.com/ibragim-bad --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/wcUJWP6WpGM-swe-rebench-lessons-from-evaluating-coding-agents-on-real-software-engineering-tasks-ibragim-bad Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/swe-rebench-lessons-from-evaluating-coding-agents-i-049bc034851a · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.