--- schema: 1 type: video title: 'How We Built an Agent That Improves Itself — Zubin Aysola, Weights & Biases' description: >- Weights & Biases' ARIA agent now helps build itself. Zubin Aysola, Senior Software Engineer on Weave, shows live how ARIA takes a production trace, turns it into an offline eval task, finds the bug (a missing SDK call), writes a fix and benchmarks the new version against the one in production. The core challenge, Aysola says, is that benchmarks, evals and agent configs all change together, so measurement has to be airtight. His team keeps the production and research agents byte-for-byte identical. They define agent variants in YAML so they can run many side by side, give the agent an unconstrained sandbox, and score it in two ways: pass/fail and relative comparison. They run 886 tasks that include simulated multi-turn users. Every production miss, and every production win, becomes a new task. That flywheel lets the team spend its time on how to make the system better instead of writing benchmarks by hand. Speaker info: LinkedIn: https://www.linkedin.com/in/zubin-aysola Related links: Weights & Biases: https://wandb.ai W&B Weave: https://wandb.ai/site/weave Timestamps: 0:00 Intro: building ARIA 0:51 Why build your own agent harness 1:31 Benchmarks, evals and agents all change together 2:06 Weave for production and offline tracing 2:56 Live demo: ARIA doing autoresearch on itself 3:40 Turning a production trace into an eval task 4:50 Nightly CI evals 5:15 Building a simulation environment 5:50 Identical agents in production and research 6:40 Generate lots of traces 7:15 A model-agnostic harness and YAML variants 8:04 An unconstrained sandbox 8:54 The eval pipeline 10:14 Scoring: pass/fail and relative 10:54 Tasks as YAML: 886 tasks 12:03 Trajectories and the eval flywheel 12:53 Demo results: ARIA fixes its own bug 14:43 Staying in the loop 16:22 Wrap-up --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/XyV6bSMyq-I-youtube-36d910cd3aa420e845df Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/XyV6bSMyq-I' markdown: 'https://liquidgold.cc/watch/XyV6bSMyq-I.md' llms: 'https://liquidgold.cc/watch/XyV6bSMyq-I/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:20:41.409Z' language: null children: [] videoId: XyV6bSMyq-I durationSeconds: 1026 publishedAt: '2026-09-26T16:00:08.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 51 endSeconds: 63 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=51' heldReason: rights - ordinal: 1 text: null startSeconds: 96 endSeconds: 107 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=96' heldReason: rights - ordinal: 2 text: null startSeconds: 107 endSeconds: 127 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=107' heldReason: rights - ordinal: 3 text: null startSeconds: 127 endSeconds: 140 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=127' heldReason: rights - ordinal: 4 text: null startSeconds: 140 endSeconds: 184 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=140' heldReason: rights - ordinal: 5 text: null startSeconds: 293 endSeconds: 304 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=293' heldReason: rights - ordinal: 6 text: null startSeconds: 358 endSeconds: 370 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=358' heldReason: rights - ordinal: 7 text: null startSeconds: 403 endSeconds: 413 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=403' heldReason: rights - ordinal: 8 text: null startSeconds: 448 endSeconds: 492 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=448' heldReason: rights - ordinal: 9 text: null startSeconds: 503 endSeconds: 536 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=503' heldReason: rights - ordinal: 10 text: null startSeconds: 536 endSeconds: 547 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=536' heldReason: rights - ordinal: 11 text: null startSeconds: 622 endSeconds: 633 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=622' heldReason: rights - ordinal: 12 text: null startSeconds: 664 endSeconds: 676 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=664' heldReason: rights - ordinal: 13 text: null startSeconds: 676 endSeconds: 729 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=676' heldReason: rights - ordinal: 14 text: null startSeconds: 729 endSeconds: 739 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=729' heldReason: rights - ordinal: 15 text: null startSeconds: 739 endSeconds: 782 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=739' heldReason: rights - ordinal: 16 text: null startSeconds: 794 endSeconds: 893 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=794' heldReason: rights - ordinal: 17 text: null startSeconds: 893 endSeconds: 904 watch: 'https://liquidgold.cc/watch/XyV6bSMyq-I?t=893' heldReason: rights --- # How We Built an Agent That Improves Itself — Zubin Aysola, Weights & Biases Weights & Biases' ARIA agent now helps build itself. Zubin Aysola, Senior Software Engineer on Weave, shows live how ARIA takes a production trace, turns it into an offline eval task, finds the bug (a missing SDK call), writes a fix and benchmarks the new version against the one in production. The core challenge, Aysola says, is that benchmarks, evals and agent configs all change together, so measurement has to be airtight. His team keeps the production and research agents byte-for-byte identical. They define agent variants in YAML so they can run many side by side, give the agent an unconstrained sandbox, and score it in two ways: pass/fail and relative comparison. They run 886 tasks that include simulated multi-turn users. Every production miss, and every production win, becomes a new task. That flywheel lets the team spend its time on how to make the system better instead of writing benchmarks by hand. Speaker info: LinkedIn: https://www.linkedin.com/in/zubin-aysola Related links: Weights & Biases: https://wandb.ai W\&B Weave: https://wandb.ai/site/weave Timestamps: 0:00 Intro: building ARIA 0:51 Why build your own agent harness 1:31 Benchmarks, evals and agents all change together 2:06 Weave for production and offline tracing 2:56 Live demo: ARIA doing autoresearch on itself 3:40 Turning a production trace into an eval task 4:50 Nightly CI evals 5:15 Building a simulation environment 5:50 Identical agents in production and research 6:40 Generate lots of traces 7:15 A model-agnostic harness and YAML variants 8:04 An unconstrained sandbox 8:54 The eval pipeline 10:14 Scoring: pass/fail and relative 10:54 Tasks as YAML: 886 tasks 12:03 Trajectories and the eval flywheel 12:53 Demo results: ARIA fixes its own bug 14:43 Staying in the loop 16:22 Wrap-up --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/XyV6bSMyq-I-youtube-36d910cd3aa420e845df Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/XyV6bSMyq-I · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.