Tim Sweeney, Principal Engineer at Weights & Biases by CoreWeave, presents ARIA, an AI research agent demonstrated live on stage running experiment batches against Karpathy's autoresearch project on real GPUs. The talk covers what ARIA does inside W&B — launching and monitoring training jobs, summarizing best runs, finding patterns across 200+ experiments, building reports and dashboards, and running in the W&B iOS app — followed by how the team built it: full trace logging to Weave, LLM judges on live traffic, tasks written as YAML unit tests, and a nightly eval suite driving go/no-go decisions. Sweeney shares production tips: invest in agent-focused observability, treat evals as CI, keep humans reviewing traces, and add value through context and tools before over-engineering the harness. # Video Identity Observations ## Observable Audience Targeting The speaker explicitly polls the live audience to segment who is present, then tailors content to each group: > "Who here... identifies as an ML researcher?... Who here is the applied engineer... Who here is in AI management? You are helping fund this compute." - **ML researchers** — those who "train models"; addressed with the auto-research loop and hypothesis collaboration. - **Applied engineers / agent builders** — repeatedly named: "Everyone here is agent builders"; a whole section is framed as "speaking to a lot of the AI agent builders in the room." - **Managers / PMs** — "let me speak briefly to the managers in the room" for productionization tips; speaker cites his PM background at Twitter's ML stack to "connect with you middle management." The agenda is explicitly designed so "each of you personas walk away with something valuable." ## Claimed Scope of the Product/Channel - **Product claim:** Weights & Biases is self-described as "the world's leading AI development platform," in business nine years, "happily joined the CoreWeave family about a year ago." - **Arya's positioning:** an "AI research and iteration agent" intended as a "data science companion" and building toward "a fully automated end-to-end research platform." - **Explicitly non-replacement:** "We're not seeking to replace RL researchers, but complement your workflows." ## Observable Presentation Patterns (demonstrated, not just claimed) - **Live-demo-driven format:** insists "This is happening live right here. This is not a fake demo," running experiments throughout the talk (live iOS announcement, live iteration, live trace inspection). - **Practitioner credibility signaling:** speaker cites "a masters in machine learning and reinforcement learning from Georgia Tech" and prior role as PM of Twitter's ML stack. - **Conference framing:** delivered at "AI Engineer Worlds Fair 2026," explaining the applied-engineer-heavy audience assumptions ("everyone here is agent builders"). - **Promotional structure:** opens with persona polling, demos product, reveals internals "because a lot of you in the audience are building agents yourself," and closes with adoption calls to action: "Please try out Arya, scan the QR codes, check out the docs." ## Self-Description Evidence - Team identity: principal engineer at Weights & Biases / CoreWeave; team follows a documented internal workflow — "We log 100% of our traces" to Weave. - Improvement methodology claimed: "We have two sort of complimentary yet adversarial research loops... to close the data flywheel." - Meta-demonstration: "We're using Arya to analyze Arya's own conversations to then make recommendations about how to improve Arya all within the UI." ## Notes - No creator claims about the channel itself (e.g., subscriber expectations, series scope) are present — this is a single conference talk; audience/scope observations apply to the session, not a broader channel. # Delivery Observation — Tim Sweeney, "An AI Research Agent That Runs Your Experiments" ## Tone & Stage Presence - **Conversational and playful.** Frequent light humor with the audience, e.g. asking who identifies as an ML researcher: *"I heard one. Wow. Okay. Great work. Great work."* - **Self-aware about marketing.** *"Man, that is a lot of uh branded buzzwords there."* - **Casual asides while demos run**, e.g. *"Yay. He I'm Arya."* after the agent introduces itself. ## Recurring Phrases & Vocabulary - **"Cooking away"** — repeated at least three times for running jobs: - *"the please summarize this project prompt is cooking away"* - *"while that's cooking"* - *"While that's cooking away, I want to show you the last item"* - **"Great" as a transition marker** — used dozens of times to pivot between segments: *"Great. But now, let's dive into the meat…"*, *"Um, great. So, that's Arya in a nutshell."* - **"Check back in / check in on"** — recurring device for the long-running live demo: *"we're going to check in on this periodically throughout this presentation"*, *"let's go ahead and check back on some of our our prompts."* - **Qualifiers and hedges**: *"sort of archetypical"*, *"pretty darn close"*, *"a little bit of a of a reduced set here."* ## Teaching Habits - **Audience segmentation by persona.** Opens with a poll (researcher / applied engineer / AI management) and repeatedly maps agenda sections back to those groups: *"hopefully each of you personas walk away with something valuable"*, *"So speaking to a lot of the the AI agent builders in the room"*, *"let me speak uh briefly to the managers in the room."* - **Roadmap-then-deliver structure.** Explicit three-part agenda announced up front, then signposted: *"quick recap here. What did Arya show?"*, *"let's just quickly recap here."* - **Explain-for-the-newcomer asides** before each product concept: *"for those who aren't familiar, [reports] is essentially a markdown file on steroids."* - **Analogy-based teaching**: *"a task as essentially a unit test for your model"*, *"tasks and evals are the new world of CI."* - **Live-demo suspense as a throughline.** Kicks off an experiment early and returns to it: *"we're going to check in on this periodically"*, resolving it near the end (*"we were right on the edge of having a live improvement, but pretty darn close"*). Also asserts authenticity: *"This is happening live right here. This is not a fake demo."* ## Communication Patterns - **Time management is spoken aloud**: *"Uh, let's see, four minutes left. Perfect."*; *"I see I'm over time, so I will click close pretty soon."* - **Announcement framing** for product news: *"announcing now today for the first time we are releasing Arya on our iOS device."* - **Pronoun inconsistency for the agent** — Arya is called both *"he"* (*"Yay. He I'm Arya."*) and *"it"* (*"we see that it identified that a new family of models emerged"*). - **Repetition/stumbling under pace**: *"the the the"*, *"are are are tracing our our prod logs"*, *"builds the bots?"* — suggests a fast, loosely scripted delivery. ## Confidence Assessment - The **persona-poll structure, "cooking away," "Great" transitions, and demo check-in device** each appear multiple times and look like stable habits. - The **pronoun inconsistency** and specific stumbles appear once or twice and may be isolated to this talk; low confidence on whether they generalize. # Video Position Record: "An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases" ## Speaker Identification - **Tim Sweeney** — self-identified in transcript as a principal engineer at Weights and Biases (and CoreWeave), holder of a masters in machine learning and reinforcement learning from Georgia Tech, former PM of Twitter's ML stack. He is the only identified speaker; all claims below are attributed to him as presenter. --- ## Section 1: Arya — Product Claims and Live Demo ### Product positioning - Sweeney introduces **Arya** as "our new AI research and iteration agent," an autonomous agent embedded in the Weights & Biases (W&B) platform. - The demo uses Andrej Karpathy's "Auto Research" project (described by Sweeney as a simple LLM-training codebase) as the grounding testbed, with **over 200 training jobs** already run in the workspace shown. ### Capabilities claimed during the live demo - **Autonomous iteration**: Arya downloaded code, set up launch jobs and GPUs, and "is able to autonomously iterate on the code itself and the hyperparameters." - **Risk-aware experiment planning**: Sweeney narrates that Arya decided "I don't want to make a big architecture swing. That feels a little bit too risky," opting instead for hyperparameter modifications. - **Live, not staged**: Sweeney emphasizes, "This is happening live right here. This is not a fake demo." - **Project summarization and pattern-finding**: Arya identified a new family of models, batch size as a high-lever parameter, and a promising architectural recipe — insights Sweeney says "would have taken me hours or days to discover on my own." - **Reporting and visualization integration**: Arya can generate W&B reports ("a markdown file on steroids"), use the parameter importance chart, and build workspaces and plots natively; Sweeney states Arya is "custom-tuned and prompted" for W&B primitives. - **iOS release**: Announced as new that day — "announcing now today for the first time we are releasing Arya on our iOS app." ### Stated positioning on automation - Sweeney frames the end goal as "a fully automated end-to-end research platform," with an explicit qualification: "we're not seeking to replace RL researchers, but complement your workflows" — Arya handles mechanics while researchers focus on new ideas and architectures. ### Live demo outcome (an honest, non-perfect result) - The live experiment batch **did not beat the prior best**: prior loss 5.831 vs. new result 5.833. Sweeney acknowledges, "we were right on the edge of having a live improvement, but pretty darn close," noting 12 experiments ran in the batch with more running overnight. --- ## Section 2: How Arya Was Built (Architecture Claims) - Described architecture: web/iOS clients → API server → queue database → worker harness, connecting to: - A **sandbox** (CoreWeave W&B sandbox) for shell calls and Python data science - An **LLM provider** (W&B inference) - **W&B Launch** + **CoreWeave GPUs** for long-running training jobs (sometimes days at a time) - **W&B Weave** for observability — "we log 100% of our traces" - Sweeney describes **"two complementary yet adversarial research loops"**: a production loop and an offline improvement loop (hypotheses → candidate agents → evals → promote via registry), framed as "closing the data flywheel." ### Weave workflow details - Agent dashboard with span/conversation volume, token tracking; a spans view showing "the topology of a trace" (tool calls, LLM calls, thinking blocks). - **Recursive self-analysis claim**: "what we're doing is using Arya to analyze Arya's own conversations to then make recommendations about how to improve Arya all within the UI." - **LLM-judge signals**: live judges flag user frustration and low-quality responses against production traffic; Sweeney shows one flagging "I'm not satisfied with the loss curve. It looks bad." - **Tasks as unit tests**: tasks defined in YAML, with two LLM judges (correctness; "actually interesting" insights — which drew laughter) and a rule-based judge (result within six tool calls). Roughly **200 tasks** run as a **nightly eval suite**. - Specific eval result cited: candidate model scored **73%** vs. prod's **72%** two nights prior — "we're definitely going to push that forward this Friday." --- ## Section 3: Productionization Tips (Speaking to Managers) Sweeney offers four prescriptive recommendations, framed as advice rather than settled findings: 1. **Invest in agent-oriented observability** — while self-admittedly biased ("I believe that weights and biases weave is the observability platform of the future"), he says any platform works; the goal is catching a new bug class he calls **"behavioral bugs"** ("Not exceptions, not performance, but behavioral bugs"). 2. **Tasks and evals are the new CI** — researchers should sit on the same scrum team as engineers, with eval metrics used as "true go/no-go decisions." 3. **Humans as a necessary judge** — a stated qualification on automation: "There are behavioral nuances that LLM will not catch. You must be using your product and... manually reviewing these traces as a team." 4. **Add value through context and tools, not harness engineering** — he warns against overengineering memory/harness: "a lot of low-hanging fruit can be ascertained through simply giving your agent context about your business domain." --- ## Notable Qualifications and Tensions Preserved - **Live demo imperfection**: The flagship autonomous experiment narrowly failed to improve on the prior result (5.833 vs. 5.831), which Sweeney acknowledged openly rather than smoothing over. - **Product bias acknowledged**: His observability recommendation favors his own company's product, which he flags himself. - **Complement, not replacement**: The full-automation vision is explicitly qualified as complementing rather than replacing researchers. - **Human-in-the-loop tension**: Despite pitching an "end-to-end" automated research platform, his productionization advice insists humans remain "a necessary judge" — a tension left unresolved in the talk. # Video Attribution & Provenance Record ## Video Overview - **Title:** *An AI Research Agent That Runs Your Experiments — Tim Sweeney, Weights & Biases* - **Video ID:** `01a15ca7-2c6e-7326-881a-a45b79256c70` - **Format:** Conference session (AI Engineer World's Fair 2026), ~20 minutes, live demo presentation ## Speaker Attribution ### Primary Speaker - **Tim Sweeney** — identified by name in the transcript - **Role:** Principal Engineer at **Weights & Biases** and **CoreWeave** - **Self-identified background:** - Master's degree in machine learning and reinforcement learning from Georgia Tech - Previously PM of Twitter's ML stack ("in a previous life was the PM of Twitter's ML stack") - Currently building Weights & Biases' agent, Arya ### Other Voices - Unidentified audience/audience reactions appear ("\[laughter\]", "\[gasps\]", "\[music\]", occasional "\>\>" speaker-change markers with no attributed names). No other named speakers occur in the transcript. ## Referenced Sources & Third-Party Claims ### Referenced Projects & Companies - **Karpathy Auto Research project** — used as the demo codebase ("a very simple project that trains an LLM"); described by the speaker as familiar to the audience. - **Weights & Biases** — company described as "the world's leading AI development platform," in business nine years, "joined the core [CoreWeave] family about a year ago." *(Company self-description — promotional framing.)* - **CoreWeave** — referenced for GPUs, sandbox, and compute infrastructure. - **Yerba Buena gardens** — mentioned as a local venue in passing. - **GLM 5.2** — mentioned as an example LLM provider a user might employ (uncertain phrasing in transcript: "GLM 5.2 two or one of your fine-tuned models"). ### First-Party Product Claims (speaker's own products — promotional) - **Arya** — the AI research/iteration agent being launched; claimed capabilities include auto research loops, workspace/report generation, run summarization, pattern finding. - **Weights & Biases Launch, Weave, Inference, Sandbox, Artifacts, Registry** — presented as components of Arya's architecture and recommended for attendees. - **iOS release announcement:** Arya released on Monday, now built into the W&B iOS app — presented as a first-time announcement ("announcing now today for the first time"). ### Numeric Claims Made in Demo - Over 200 training jobs in the demo project; 12 experiments run in the live batch. - ~200 tasks clustered into a nightly eval suite. - Candidate model scored **73%** vs. production model's **72%** two nights prior; promotion planned for Friday. - Live experiment result: 5.833 vs. prior best 5.831 ("right on the edge of having a live improvement"). ## Promotional Segments (flagged) - The entire session functions as a product presentation for Weights & Biases/CoreWeave; the speaker explicitly invites attendees to: - "try out Arya, scan the QR codes, check out the docs" - "try coreweave weights and biases sandbox" - "use weights and biases inference" - Audience-facing tips conclude with a recommendation of W&B Weave as "the observability platform of the future" — flagged as a promotional self-endorsement. ## Audience Interaction (provenance note) - Speaker polled the audience for ML researchers, applied engineers, and AI management roles; audience responses were verbal and not individually attributed. ## Availability & Access Limits - No paywalls, registration requirements, or restricted content are mentioned. - Comment availability: not stated in transcript. - No external citations, papers (beyond the Karpathy project reference), or archival sources are cited. ## Uncertainty Notes - Speaker names himself "Tim Sweeney"; no independent verification of identity or titles is available in the transcript. - Transcript contains transcription artifacts (e.g., "turn database" possibly "Torn/Qdrant-like database," "eating at lunch" vs. "done at lunch"); exact wording of some demo UI content is approximate. - The "\>\>" markers indicate speaker changes but no second named speaker is ever identified. Okay. Hello everyone and thank you for attending this session. My name is Tim Sweeney, a principal engineer at Weights and Biases and Coreweave. And for the next 20 minutes, we're going to talk about Arya, our new AI research and iteration agent. Let's go ahead and get started. So, uh, first off, just by way of making some noise, some clapping, uh, who here, um, identifies as an ML researcher? You're someone that trains models, trains the brain? I heard one. Wow. Okay. Great work. Great work. Uh, what about who here is the applied engineer, the namesake of this conference? Who here actually builds the bots? \>\> Okay, good. Expected much more. And who here is in AI management? You are helping fund this compute. Okay. Okay. Nice. From the back. Lovely. Um, well, now that I know a little bit about you, just a little bit about me. Uh, again, my name is Tim. I have a masters in machine learning, uh, and reinforcement learning from Georgia Tech. So, I've been that, uh, researcher currently building Weights and Biases agent, Arya. So, identify as that applied engineer. And in a previous life was the PM of Twitter's ML stack. So, I hope you hopefully can connect with you middle management as well. \[laughter\] \>\> \[gasps\] \>\> Um today's agenda is kind of broken into three sections and hopefully each of you personas walk away with something valuable. So first we're going to learn about Arya itself and how it can supercharge your AI and ML workflows. We're going to dive into auto research and see that live in a live demo in just a moment. Then we're going to pull back the curtain and learn how we use weights and biases and uh coreweave to actually build Arya because a lot of you in the audience are building agents yourself and we believe a lot of these components can help you in your endeavors. And then towards the end we'll just take a step back and identify a few key tips and tricks for making sure that you're able to productionize your systems effectively. For those of you who might not be familiar, Weights and Biases is the world's leading AI development platform. We've been in business now for nine years and have happily joined the core family about a year ago. Uh we have a number of products in our suite but are really known for our models training inference and weave stack which really helps collect data uh about the AI development and machine learning workflows and makes that information actionable and uh enables users to make the best decisions about what to do next. So without further ado, let's go ahead and dive into Arya, our agent. Uh we'll show a demo and then we'll get back to some slides. Okay, beautiful. Let's make this a bit bigger. Holler at me if you need it to be bigger. So, uh, what you're looking at here is a weights and biases workspace. For you, for anybody that isn't familiar, on the lefth hand side, I actually see a list of a bunch of different experiments. In this particular project, I have over 200 training jobs. And on the right hand side, I see a scatter plot of, in this case, declining metrics, which is good. means our loss is going down over time. And this view would be very familiar for anyone that uses our tool. Now, to ground this, we're actually uh uh using the Carpathy Auto Research project, which I'm sure many of you are familiar with, but if you're not, it's just a very simple project that trains an LLM, and it's a great foundation for auto research type demonstrations because it's a very simple codebase and allows us to improve iteratively over time. So, let's jump back to the project and open up Arya by clicking this blue button in the upper right. When I click this button, I'm uh presented with the familiar chat interface with, you know, how can I help you today, a few call to actions, and you know, I can add different context in my project or maybe add images, etc. Um, everyone here is agent builders, so I don't need to bore you with the details of what an agent interface looks like. But let's go ahead and just, you know, enter in a basic intro here. Let's say, "Hello, Arya. You're on stage at AI World's Fair 2026. Please introduce yourself." So, it's going to go ahead and chug along and hopefully emit some sort of nice emoji. Yay. He I'm Arya. I'm talking to the audience. Great. But now, let's dive into the meat of why you came here. So, I'm going to open up this chat here. And this is a longunning chat where I've been running again over 200 experiments using the auto research loop. Um, it helped me download the code, set up my launch jobs, set up my GPUs, and is able to autonomously iterate on the code itself and the hyperparameters. We'll take a look at what it's doing in a moment, but while we're doing this, I'm going to kick off a live iteration right here. So, what I'm going to say is please conduct another batch of experiments. You are on stage at the AI Engineer Worlds Fair 2026 and we're hoping to find the best model live. I believe in you. Uh, because we know we have to encourage our models. Um, so it's been doing this for a while. What it what it's doing here is it's saying, "Okay, great. Um, I don't want to make a big architecture swing. That feels a little bit too risky." So, it's probably going to go for uh some modifications to the hyperparameters. And then it's kicking off a shell call here that is actually um executing that uh executing that experimentation loop. And we're going to check in on this periodically throughout this presentation. But I want to help explain what's going on behind the scenes. So behind the scenes I have set up a weights and biases launch queue. Launch is our our product that allows you to connect your compute clusters and allows humans and agents to launch longunning experimentation jobs particularly by leveraging GPUs. Here I'm looking at a uh a terminal output of my Kubernetes cluster where we're actually seeing live execution of experiments happening. So this is happening live right here. This is not a fake demo. Um great. And if we jump back, we see that at this point it started the cues and now it is simply polling and waiting for our work to be complete. So we'll jump back to that in a in a moment. But before but let's dive into a few other examples. So uh something else that is interesting you can do is maybe you might want to ask it something like please summarize the highest performing runs in this project. This use case would be something like maybe a new user come or a new uh team member is joining your project and want to understand the research. Um or maybe you've uh someone's been doing some work while you were on PTO and you want to get caught up. We'll see what this comes up with in a moment. Some other pre-anned uh examples are finding patterns in your project. So here we can see that I asked it, hey, can you find some patterns in this research? And we see that um it identified that a new family of models emerged as the as the auto uh auto research was happening. Uh it identified that batch size seems to be a really high high uh lever uh parameter. It identified an architectural recipe that seemed to be quite promising and a number of other insights that would have taken me hours or days to discover on my own. And Arya is able to do it right for me directly in the interface that I already live. Not only is it able to emit text based uh textbased outputs, but it also deeply integrates with a number of weights and biases visualization utilities. So here I've actually asked it to emit a weights and biases report which for those who aren't familiar is essentially a markdown file on steroids. It's got uh embedded embedded plots, charts and and and graphics. And so here uh you know it's talked about the thesis of the project. It's it's emitted a number of of data panels. And uh I actually think it's quite interesting. It used um one of our more esoteric panels, the uh parameter importance chart to uh tell me the correlation of various different parameters within this uh within this training job. Uh in addition to uh reports, it's also great at working with workspaces. So if you're a weights and biases user, uh you spend a lot of your time uh designing and working with workspaces. Well, Arya is actually customtuned and prompted to really understand how to build workspaces, build plots, and complement that that data analytics with real live graphics using the built-in proprietary charts that weights and biases users know and love. Um, so with that, let's go ahead and check back on some of our our prompts. We can see that the please summarize this project prompt is cooking away. It's querying weights and biases. It's applying patches. It's writing its own code. So, we'll come back and check on that in a moment. and our longunning training job is uh still pulling for the results. We can see that we're cooking away on our GPUs. So, we're we're frying some GPUs \[music\] and doing some data science all live. And while that's cooking, let's go ahead and jump back to the presentation. We'll come back in a moment. Uh oh, no, we're not looking at a dictionary. We're looking at a PO. Great. Uh okay, so quick recap here. What did Arya show? What did we show in these last five minutes? First, we show that uh Arya can serve as your data science companion right inside of Weights and Biases, helping you discover insights that you wouldn't you wouldn't be able to discover as your experiments and as your team size grows. Next, we address the problem of complicated reporting and complicated plotting. Weights and biases users are are really want to turn their insights into visual communication tools. They want to communicate with their peers and their colleagues. So Arya's built from the ground up to understand those primitives and help co-pilot and drive right along right alongside in the UI and announcing now today for the first time we are releasing Arya on our iOS device or on our iOS app. So uh uh Arya released on Monday and our iOS app now has Arya built in. So if you're conducting hyperparameter tuning jobs, if you're training models, or if you're just researching within the weights and biases ecosystem, you can go touched grass at Yerba Buena uh gardens and steer your uh hyperparameter tuning jobs all from your mobile device. And what is this all building up to? This is building up to an a fully automated endto-end research platform where we're not seeking to replace uh RL researchers, but complement your workflows. Arya's great at orchestrating jobs, understanding GPU workloads, responding to events within the within the Wandi ecosystem, and listening to researchers, uh, uh, looking up archive papers, and collaborating on hypothesis. So, we can let Arya drive the mechanics that you don't want to deal with while you focus on the new ideas, new architectures, and new parameters that you wanted to try. Um, great. So, that's Arya in a nutshell. We're really hoping that you give it a shot. And uh we'll jump back to the auto research at the end and see if we got a new best record. But before we do that, let's talk about how we use weights and biases and coreweave to actually build Arya. So now speaking to a lot of the the AI agent builders in the room, here's a quick architecture on the lefth hand side. You see that we have a web client, iOS client that communicates with our API server that then dumps data into our turn database and is worked on by our harness, our our worker harness. This is sort of archetypical of probably what most of you are all building in the room and is exactly what we have on our back end. But that harness worker is a magic is a is a magic box and it connects to a number of important utilities. First is a sandbox where it can execute arbitrary shell calls uh do do Python data science etc. And we invite you to try coreweave weights and biases sandbox to fit into your architecture. Next up you need an LLM provider of course and so if you're maybe using GLM 5.2 two or one of your fine-tuned models. We invite you to use uh weights and biases inference and connect that to your worker as well. If you're like us, you need to run longunning workloads outside of the main loop of the agent where you're actually training for day for sometimes days at a time. Weights and biases launch can actually help facilitate that and coreweave GPUs can help make that compute even better. And then lastly, and really most importantly, we need an observability layer. It's critical that your agents are able to log out their what's going on with their sessions, their turns, their tool calls, any errors they're hap that that's happening, etc. Uh we have a product called Weights and Biases Weave that we log 100% of our traces to where us and our team can learn from. And that's where we move from production to offline where our team is able to use Weights and Biases Weave to drive insights and identify behaviors, implement tasks with tasks which are essentially unit tests for your models and evaluate those models in a loop. We have a model repository which you might choose to use weights and biases artifacts to store your agents or models and you we emit our evaluation results to weave where we have a common dashboard that we can make go no-go decisions on various prompt changes or architectural changes that then feeds into a research loop which we call our improvement loop where we form hypotheses implement candidate agents and analyze the evals. So we have two sort of complimentary yet adversarial research loops going on going on offline feeding data from weights and biases weave ultimately to identify the best model so that we can promote that to production through our registry and close the data flywheel. So in the next just uh three seven minutes or so we'll just talk about uh weights and biases weave and show how we as a team actually use weave to facilitate this workflow and we believe this is something that you would benefit from as well all of you agent builders in the room. Yes, another demo. Great. Okay. Okay, we have new responses. So, it's going to be exciting when we open this up later. See if uh we've got some better metrics. Um, okay. Let me zoom out just a little bit here. So, here I'm looking at the agent dashboard. This is the live weights and biases agent or Arya agent dashboard uh built in weave. Man, that is a lot of uh branded buzzwords there. This is the dashboard that you would get if you use our tool. and uh you have a you know uh span volume, conversation volume, token tracking, etc. Think of this as like a uh a bird's eye view of your agent. For me, however, I really like this conversations view, which I do have pre-loaded in this tab. This conversations view is a live feed of all of the conversations that are going through Arya, but it's filtered down to just the internal employees. So, it's a little bit of a of a reduced set here. Um what I what I love is this middle spans view which gives me a visual indicator of the topology of a trace. Different colors and and shapes indicate different things that are happening within the agent. So things like tool calls, LLM calls, thinking blocks, etc. which really help me understand again the shape and topology of that particular conversation. I can of course open up one of these conversations and view our our conversation view where I can see the system prompt, the user message, shell calls, reasoning blocks, etc. This is where my research lead, myself and my PM go to add notes, add feedback, add emojis, and talk about and discover those insights and those behavioral nuances we spoke about earlier so that we can turn them into tasks. Arya's built in to the weights and biases system as well. Here you'll see a summarize button and these are sprinkled throughout the weights and biases application. I simply click summarize and we start a new chat contextualized to the thing that I'm looking at. So it it sees this and says give me a brief summary of this particular conversation. So if you if you're paying a attention closely, you'll realize that what we're doing is using Arya to analyze Arya's own conversations to then make recommendations about how to improve Arya all within the UI. Um okay, great. While that's cooking away, I want to show you the last item uh within the Weave ecosystem here, and that's signals. We've heard a lot today about the value of evals and the value of LLM judges. Weave actually offers an integrated LLM judge experience. So here, if I zoom out a little bit, you'll see that I have a user frustration signal, a lowquality response signal, ask user signal, etc. These are LLM judges that run live against against our live traffic. And we can see various different signals like user frustration moments or lowquality responses. These help our team identify these clusters of behavior for us to go fix in next week's iteration. Let's go ahead and do a live look and see what it says. Um this says the user explicitly states that I'm not satisfied with the loss curve. It looks bad and it apparently that indicates frustration. So here we can see an LLM judges live reasoning for why that particular flag was uh indicated. Uh let's see, four minutes left. Perfect. Um so, uh with that, I've been using the term task a lot. And so what we're do, what I've showed so far is is this live production loop where we are are are are tracing our our prod logs. We're looking at them as humans, maybe even using LLMs to complement that analysis. And what we end up doing is transforming those into tasks. Now, this gets a bit technical here, but our tasks are all described as YAML files. You can think of a task as essentially a unit test for your model. So here we say we have a an example user prompt that says check this run and that run. Both of these are giving good results. What can we learn from this? What's the difference? So this is an example of something we want Arya to be good at for all of you. And after the uh requisite metadata we see that we've defined an LLM judge. So here we've defined what correctness means in the context of that question. And we've then we've defined a second LLM judge that determines if the insights are actually interesting. \[laughter\] And then we've uh defined a third rule-based judge that says were you able to actually generate a result within just six tool calls meaning it got there with some degree of expediency. These are all then clustered together into we have about like 200 of these. They're all clustered together into an eval suite that runs nightly. And again we use weave to track all those evals. So here, I know it's a bit small on this screen, but what you're looking at is a listing of every night's eval. This is literally two nights ago, the evaluation for our candidate model got 73% on our production or on our eval suite against the 72% that our prod model got, which means we're definitely going to push that forward uh this Friday. Uh and we can see a kind of a a performance plot on the right. So these utilities are what you would get out of the box if you're uh if you decide to pick up weave and use this tool. Um, jumping back to the last conversation we had where it asked me where we asked, uh, can you please give a quick summary of this trace, we see that it actually analyzed the conversation, understood what the user was doing, and then ultimately decided that this was a pretty strong trace. Um, let's see, we've got two and a half minutes left, so let's just quickly recap here. Uh, first off, uh, what we use weave to do is a, collect production traffic. Super critical to collect all of your production traffic so you can learn and iterate. Secondly, we use it to generate insights both as humans as well. We we do it as humans. We use Arya and we use LLM judges to identify those behavioral nuances. We then enrich our tasks. We implement models and we evaluate using weights and biases weave as a shared dashboard where we can make decisions together as a team that then ultimately allows us to promote the best model forward with confidence. So speaking of confident productionization, let me speak uh briefly to the managers in the room. So a few tips for being successful here. First is um invest in agent-oriented observability. Uh I'm a bit biased. I believe that weights and biases weave is the uh observability platform of the future. Uh but pick your favorite flavor. Whatever it is, log your sessions, log your turns, log your tools and feedback. This introduces an ability to catch a new class of bugs in our world called behavioral bugs. Not exceptions, not performance, but behavioral bugs. Next up, tasks and evals are the new world of CI. You've heard a lot about this. If you are a software engineer, you've written unit tests your whole life. You must develop a practice where your researchers are sitting on the same scrum team as you developing tasks and you're viewing the performance metrics as true go no-go decisions. But in order to complement that, you must use humans as a necessary judge. There are behavioral nuances that LLM will not catch. You must be using your product and you must be manually reviewing these traces as a team at the end of the week on a board looking at the best and worst traces to understand how your model is performing. And then lastly, um just maybe one one more tip is to add value through context and tools. It can be really tempting to uh try to overengineer the harness and do a bunch of creative stuff around memory and things like this. We found that a a lot of lowhanging fruit can be ascertained through simply giving your agent context about your business domain, the underlying uh primitives that you have available and your particular uh business data. Um so with that, let's go ahead and check in on our uh our our research agent here and let's go ahead and toggle our workspace. And what we should be seeing is yes indeed a little dot that uh oh okay our previous dot which was done at lunch was 5.83. 831. This got 5.833. So we were right on the edge of having a live improvement, but pretty darn close. Uh so that's what the uh that's what the model was able to produce. It actually uh ran uh quite a few tests here. I see I'm over time, so I will click close pretty soon. But we ran 12 different experiments within that experiment batch and uh we'll be running more all night. So please try out Arya, scan the QR codes, check out the docs. Uh we really love to see what you do with it and um looking forward to serving you. Thank you very much. \>\> \[music\]