--- schema: 1 type: video title: >- From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases description: >- An agent repairs a broken git repository and earns a perfect score, but the run stays out of the default leaderboard because it tested only one task. Wolfram Ravenwolf uses that distinction to build an evaluation pipeline where every score can be traced back to its conditions. WolfBench compares the average, best and worst runs, the tasks solved at least once, and the tasks solved every time. Repeating the same configuration exposes the gap between a lucky success and dependable behavior. The configuration includes six layers: benchmark, runner, sandbox, agent, model service and settings. Versions, provider routing, system prompts, resource limits and timeouts all affect what the result actually measures. The live workflow launches a Terminal Bench task through Harbor, collects the results, filters incomplete runs in a Marimo dashboard, and turns valid runs into an interactive leaderboard. Weave preserves the trajectories behind the bars so reviewers can inspect commands, messages and failures. Ravenwolf examines two traps in interpreting the results: a cheaper model can consume enough tokens to raise total cost, and greater reasoning effort can lower the score when it leaves too little time to execute commands. Infrastructure failures need different treatment from genuine task failures, while zero temperature still does not guarantee identical runs. The resulting comparison helps teams choose a configuration for their own constraints, with costs and consistency visible alongside the headline score. Speaker info: - https://www.linkedin.com/in/wolframravenwolf/ - https://x.com/WolframRvnwlf - https://wolfbench.ai/ - https://github.com/wandb/WolfBench Timestamps: 0:00 - Workshop setup 0:52 - Meet Wolfram Ravenwolf 3:31 - Evaluation mindset and methodology 4:12 - Why one average hides reliability 5:48 - The five WolfBench metrics 6:30 - Repeat runs under uniform conditions 7:24 - Evaluate a configuration, not just a model 9:53 - Six layers of an evaluation pipeline 10:50 - Choose and pin the benchmark 12:37 - Harbor and evaluation runner versions 14:31 - Isolated sandboxes 16:37 - Compare agent harnesses 18:38 - Settings that change the result 20:00 - Prompts, tools and resource limits 21:33 - Timeouts, concurrency and retries 25:35 - Fix inference provider routing 27:39 - Launch the git recovery task 30:41 - Preserve evidence beyond the score 31:49 - The WolfBench collection toolkit 34:19 - Inspect runs in the Marimo dashboard 34:59 - Filter incomplete and invalid runs 36:09 - Export results and generate charts 36:50 - A perfect score on one task 37:18 - Read the interactive leaderboard 39:06 - Consistency, average and ceiling 41:22 - Token usage and total cost 45:24 - When more reasoning lowers the score 47:39 - Inspect the underlying Weave traces 49:13 - Review configurations and failure causes --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/YIxzNZDDy1A-youtube-561cf6576bf9b94c7ffc Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/YIxzNZDDy1A' markdown: 'https://liquidgold.cc/watch/YIxzNZDDy1A.md' llms: 'https://liquidgold.cc/watch/YIxzNZDDy1A/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:19:14.309Z' language: null children: [] videoId: YIxzNZDDy1A durationSeconds: 3150 publishedAt: '2026-10-11T17:00:18.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 12 endSeconds: 56 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=12' heldReason: rights - ordinal: 1 text: null startSeconds: 67 endSeconds: 221 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=67' heldReason: rights - ordinal: 2 text: null startSeconds: 232 endSeconds: 254 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=232' heldReason: rights - ordinal: 3 text: null startSeconds: 254 endSeconds: 266 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=254' heldReason: rights - ordinal: 4 text: null startSeconds: 266 endSeconds: 355 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=266' heldReason: rights - ordinal: 5 text: null startSeconds: 355 endSeconds: 365 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=355' heldReason: rights - ordinal: 6 text: null startSeconds: 365 endSeconds: 398 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=365' heldReason: rights - ordinal: 7 text: null startSeconds: 409 endSeconds: 452 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=409' heldReason: rights - ordinal: 8 text: null startSeconds: 452 endSeconds: 463 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=452' heldReason: rights - ordinal: 9 text: null startSeconds: 463 endSeconds: 596 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=463' heldReason: rights - ordinal: 10 text: null startSeconds: 607 endSeconds: 660 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=607' heldReason: rights - ordinal: 11 text: null startSeconds: 660 endSeconds: 671 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=660' heldReason: rights - ordinal: 12 text: null startSeconds: 671 endSeconds: 761 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=671' heldReason: rights - ordinal: 13 text: null startSeconds: 761 endSeconds: 772 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=761' heldReason: rights - ordinal: 14 text: null startSeconds: 873 endSeconds: 885 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=873' heldReason: rights - ordinal: 15 text: null startSeconds: 1011 endSeconds: 1123 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=1011' heldReason: rights - ordinal: 16 text: null startSeconds: 1123 endSeconds: 1134 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=1123' heldReason: rights - ordinal: 17 text: null startSeconds: 1134 endSeconds: 1210 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=1134' heldReason: rights - ordinal: 18 text: null startSeconds: 1210 endSeconds: 1221 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=1210' heldReason: rights - ordinal: 19 text: null startSeconds: 1221 endSeconds: 1298 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=1221' heldReason: rights - ordinal: 20 text: null startSeconds: 1298 endSeconds: 1308 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=1298' heldReason: rights - ordinal: 21 text: null startSeconds: 1308 endSeconds: 1540 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=1308' heldReason: rights - ordinal: 22 text: null startSeconds: 1551 endSeconds: 1666 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=1551' heldReason: rights - ordinal: 23 text: null startSeconds: 1677 endSeconds: 1848 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=1677' heldReason: rights - ordinal: 24 text: null startSeconds: 1858 endSeconds: 1914 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=1858' heldReason: rights - ordinal: 25 text: null startSeconds: 1914 endSeconds: 1926 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=1914' heldReason: rights - ordinal: 26 text: null startSeconds: 2059 endSeconds: 2070 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=2059' heldReason: rights - ordinal: 27 text: null startSeconds: 2117 endSeconds: 2175 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=2117' heldReason: rights - ordinal: 28 text: null startSeconds: 2186 endSeconds: 2224 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=2186' heldReason: rights - ordinal: 29 text: null startSeconds: 2258 endSeconds: 2356 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=2258' heldReason: rights - ordinal: 30 text: null startSeconds: 2486 endSeconds: 2496 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=2486' heldReason: rights - ordinal: 31 text: null startSeconds: 2726 endSeconds: 2736 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=2726' heldReason: rights - ordinal: 32 text: null startSeconds: 2860 endSeconds: 2873 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=2860' heldReason: rights - ordinal: 33 text: null startSeconds: 2960 endSeconds: 2972 watch: 'https://liquidgold.cc/watch/YIxzNZDDy1A?t=2960' heldReason: rights --- # From Zero to Leaderboard: Agent Evaluation — Wolfram Ravenwolf, Weights & Biases An agent repairs a broken git repository and earns a perfect score, but the run stays out of the default leaderboard because it tested only one task. Wolfram Ravenwolf uses that distinction to build an evaluation pipeline where every score can be traced back to its conditions. WolfBench compares the average, best and worst runs, the tasks solved at least once, and the tasks solved every time. Repeating the same configuration exposes the gap between a lucky success and dependable behavior. The configuration includes six layers: benchmark, runner, sandbox, agent, model service and settings. Versions, provider routing, system prompts, resource limits and timeouts all affect what the result actually measures. The live workflow launches a Terminal Bench task through Harbor, collects the results, filters incomplete runs in a Marimo dashboard, and turns valid runs into an interactive leaderboard. Weave preserves the trajectories behind the bars so reviewers can inspect commands, messages and failures. Ravenwolf examines two traps in interpreting the results: a cheaper model can consume enough tokens to raise total cost, and greater reasoning effort can lower the score when it leaves too little time to execute commands. Infrastructure failures need different treatment from genuine task failures, while zero temperature still does not guarantee identical runs. The resulting comparison helps teams choose a configuration for their own constraints, with costs and consistency visible alongside the headline score. Speaker info: - https://www.linkedin.com/in/wolframravenwolf/ - https://x.com/WolframRvnwlf - https://wolfbench.ai/ - https://github.com/wandb/WolfBench Timestamps: 0:00 - Workshop setup 0:52 - Meet Wolfram Ravenwolf 3:31 - Evaluation mindset and methodology 4:12 - Why one average hides reliability 5:48 - The five WolfBench metrics 6:30 - Repeat runs under uniform conditions 7:24 - Evaluate a configuration, not just a model 9:53 - Six layers of an evaluation pipeline 10:50 - Choose and pin the benchmark 12:37 - Harbor and evaluation runner versions 14:31 - Isolated sandboxes 16:37 - Compare agent harnesses 18:38 - Settings that change the result 20:00 - Prompts, tools and resource limits 21:33 - Timeouts, concurrency and retries 25:35 - Fix inference provider routing 27:39 - Launch the git recovery task 30:41 - Preserve evidence beyond the score 31:49 - The WolfBench collection toolkit 34:19 - Inspect runs in the Marimo dashboard 34:59 - Filter incomplete and invalid runs 36:09 - Export results and generate charts 36:50 - A perfect score on one task 37:18 - Read the interactive leaderboard 39:06 - Consistency, average and ceiling 41:22 - Token usage and total cost 45:24 - When more reasoning lowers the score 47:39 - Inspect the underlying Weave traces 49:13 - Review configurations and failure causes --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/YIxzNZDDy1A-youtube-561cf6576bf9b94c7ffc Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/YIxzNZDDy1A · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.