Will Brown (Prime Intellect) on applying reinforcement learning to tasks that lack automatically checkable answers. He recaps RL fundamentals, explains why verifiable rewards break down outside domains like math and code, and presents techniques for building reward signal without ground truth: judge models, question-answer pairs grounded in real documents and repos, and a "reverse direction" approach where hidden elements (e.g., a bug or backdoor) are inserted so the model learns to find them—also serving as a difficulty dial. He covers reward hacking risks and mitigation via trace inspection and expert input, and argues for open models and shared benchmarks to make RL a science. Talk delivered at the AI Engineer conference. # Channel Identity Observations — RL Without Verifiable Rewards (Will Brown) ## Speaker & Explicit Self-Description - **Self-identified role:** "I'm Will Brown. I lead applied research at Primordial AI" (note: he also refers to the company as "Prime Intellect"/"Primary and Like"/"Prime and Elect" in the transcript — name rendered inconsistently by transcription). - **Explicit company scope claim:** > "We kind of go both from the we start at the compute layer and do lots of large-scale GPU orchestration. We build the Primary RL training framework... We build environments... We have a training platform called Lab." - **Claimed mission:** > "The goal that we're really trying to enable is more people to be able to become their own research lab, become... take ownership over the intelligence of their own model weights." - **Commercial offer (explicit):** "We work hands-on with startup enterprises to help them train their models. We are also hiring quite a lot." ## Intended Audience - **Conference attendees in AI engineering / post-training**, with a crash-course option for newcomers: > "I would imagine if you're in the post training session here you've probably heard a little bit about RL but for those of you at home and... looking for the crash course." - **Practitioners deploying agents in production** — heavy focus on production traces, deployed agents, and enterprise training. - Technical sophistication assumed: listeners are expected to know GRPO, RLVR, SFT, on-policy distillation, prompt optimization (DSPy referenced as "Jeph I"), reward hacking. ## Claimed Scope of the Channel/Creator's Work - Full RL/post-training stack: compute orchestration → training framework → environments (tasks, harnesses, verifiers) → hosted platform ("Lab") for training, evals, inference. - Positioning around **open-source base models**: "the models are starting generally from some open-source base model, and you're optimizing it for your task." - Research + product: "synthesizing a lot of work we've been doing as well as from the broader research literature" — references own blog posts ("general agent") and external collaboration ("Echo from some researchers that we are friends with"). ## Observable Presentation Patterns (Demonstrated, Not Claimed) - **Pedagogical style:** opens with plain-language recap of RL fundamentals (agent = model + harness; environment = task + world) before advanced material — accessible entry point. - **Core thesis framing:** verifiable rewards (math, code, tool use) are the "easy case"; "most real-world tasks are not this verifiable" — the talk is structured around techniques to "manufacture signal": grounding, judges, scaled search, world simulators, hindsight rubrics, difficulty calibration. - **Named recurring concept:** "continual learning" defined specifically as deployed models learning from mistakes post-deployment with "humans in the loop at that right level of abstraction." - **Honest hedging about the field's maturity:** > "We don't know how it all works and we're figuring it out as we go in many cases." > "Today we still do [fuss with research pieces], but we're kind of seeing paths forward." - **Business-forward close:** ends with explicit enterprise/hiring pitch and invitation to follow up in person. ## Claims vs. Demonstrated — Distinction Notes - **Creator claims:** effectiveness of their tooling, simulator fidelity, benchmark uplifts from "general agent" blog (stated, not demonstrated in transcript). - **Demonstrated:** deep technical fluency in RL mechanics, reward hacking, judge-based evaluation; a synthesis-plus-product narrative typical of a vendor-affiliated applied researcher speaking at an industry conference (AI Engineer, post-training session). # Delivery Observations: Will Brown — "Reinforcement Learning without Verifiable Rewards" ## Tone & Register - **Conversational, low-formality conference talk style**: Frequent filler words throughout — "Uh thanks all for coming," "Um hopefully lots of fun stuff today," "Um but often we don't actually have verifiable rewards." The delivery is casual rather than rehearsed-polished. - **Inclusive, audience-aware framing**: He repeatedly addresses mixed expertise levels: "for those of you at home and uh for those uh who are kind of still just kind of looking for the crash course" — acknowledging not everyone in the room shares background knowledge. - **Candid admissions**: Openly states uncertainty about the field, not just his content: "We don't know how it all works and we're figuring it out as we go in many cases." ## Vocabulary & Terminology Habits - **Heavy reliance on the qualifier "kind of"**: This is the single most frequent verbal tic. Representative examples: - "we can kind of look at what they did" - "you can just kind of like check this deterministically" - "it's easier to tell what went wrong in hindsight. And so, the fact that you've already seen the chain of events after and you can look backwards" - **Recurring concept-pair vocabulary**: "messy real world tasks," "messy situations," "messier environments" — the word **"messy"** is his signature descriptor for non-verifiable settings. - **Agent-perspective framing with "we"**: Consistently uses "we" for both the research community and his company: "we've been doing," "we've found," "we've spent a lot of effort." ## Teaching Habits & Structural Patterns - **Definition-by-decomposition**: He explains concepts by breaking them into named components: - > "an agent which we're going to call a model plus a harness which we place into an environment. And so an environment we're going to call a task plus a world." - He even pre-defines jargon before using it: "The policy here is just the model weights themselves." - **Easy-case → hard-case contrast structure**: A repeated rhetorical pattern — first establishing what's tractable, then pivoting with "but": - "these are kind of the easy cases where the reward design problem is not so difficult. Um but most real-world tasks are not this verifiable." - "RL's great for refining skills, but less so for incorporating like dense new knowledge." - **Concrete instantiation of abstractions**: Follows each abstract claim with concrete examples ("maybe it's a docker image maybe it's a code base maybe it is a collection of uh task specific tools"). - **Informal quotation of internal reasoning**: Uses embedded pseudo-dialogue to explain concepts: - > "saying, 'Okay, how do I make the model do things that have higher reward?'" - > "'Okay, the model made a mistake here. This thing doesn't feel quite right.'" - **Explicit jargon hedging**: He defines buzzwords on his own terms before using them: "continual learning, which is a big buzzword that I think a lot of people like talking about in many different ways, but I'm going to use it to mean a very particular thing." ## Recurring Thematic Phrases - **"Figuring it out as we go"** — appears twice for the same concept (lines ~35 and ~234), functioning as a thematic throughline. - **"Spending compute"** as a repeated formula: "spend computes to make decisions," "you can spend compute on searching these," "spend compute mining these traces," "spend inference compute on refining." - **"Working backwards"** as a named teaching principle: "This general principle of like working backwards is starting from the solution... You can verify the easy problem and then learn on the hard problem." ## Time & Pace Management - He explicitly signals time pressure near the end: "I know we're running a little short on time, so I wanted to break up a couple blogs" — suggesting normally paced, slightly overlong material compressed on the fly. - Ends with a standard conference close: hiring/partnership plug — "We are also hiring quite a lot. If you want to get in touch for either of these, find me after the talk." ## Uncertainty Notes - The "kind of / sort of" frequency is estimated from representative examples; no exact count was performed. - Company-name inconsistency ("Primordial AI," "Primary and Like," "Prime and Elect") appears in the transcript and likely reflects transcription error rather than speaker behavior — delivery conclusions should not be drawn from it. # Observation Document: "Reinforcement Learning without Verifiable Rewards" — Will Brown ## Metadata - **Speaker**: Will Brown, introduced as leading applied research at Prime Intellect (introduces himself as "Primordial AI" — note: transcript contains both names; company referred to later as "Prime Intellect" / "Primary and Like" [likely transcription artifacts]) - **Venue**: AI Engineer conference, post-training session - **Context**: Talk synthesizing Prime Intellect's work and broader research on extending RL beyond verifiable rewards ## Core Thesis - **RLVR dominates current thinking**, but many real-world tasks lack verifiable rewards — success is often judged in hindsight rather than specified up front. - Messy tasks require **manufacturing signal** through grounding, judges, search, simulators, and test-time compute. ## Stated Positions & Reasoning ### RL Fundamentals (as framed by Brown) - Agent = **model + harness**; environment = **task + world + scoring rule**. - All popular RL algorithms (GRPO, REINFORCE, SISPO) sit in the **policy gradient framework**: create an advantage (reward minus baseline, possibly scaled) and nudge weights toward higher reward. ### Environments and Evals - **"Environments and evals are really the same thing"** — the same objects support SFT data generation, RL, on-policy distillation, prompt optimization (e.g., "Jeph I" — likely transcription of a prompt-optimization method), and agent iteration. ### Verifiable vs. Non-Verifiable Rewards - **Easy cases**: math (parse boxed answers), code (test cases, linters), tool use (deterministic database-state checks). - **Hard cases**: report writing, booking flights, "handling a refund" — no clean best answer; signal is unclear. ### Why Evals Are Hard - Benchmarks are typically **a few hundred tasks handcrafted over months** with experts and data vendors — not scalable. - Real-world distributions are **unbounded**; "we are figuring out the distribution as we go," and Brown notes classical ML treats out-of-distribution generalization as undefined. - **Reward hacking**: loose proxies undefined at boundaries; RL exploits them. ### Goal: Continual Learning - Brown's definition: models deployed in messy settings that **make mistakes, catch them after the fact, and avoid repeating them**. - Current abstraction level for human oversight is "far too low to be practical for most people"; many difficult processes are automatizable "they just are kind of difficult problems to solve." - RL refines skills but is "less so for incorporating dense new knowledge" — argues for blending RL with supervised learning from the environment. ### Techniques for Manufacturing Signal - **Grounding**: A/B testing with/without source material creates a capability gap exploitable as signal. - **Judges**: LLMs are powerful general reasoners; set up correctly, they can spend compute judging actions. - **Working backwards**: start from a known solution/reachable end state, throw it away, learn to find it — "supervision for free." Examples: doc-derived QA pairs; code PRs broken into replayable pieces. - **Production traces**: deployed-agent traces define the task distribution before labels exist — "tells us at least what we want to look for." - **World simulators**: for tools/apps without backend controllability (MCP tools, CLIs, websites), build simulators via backend infrastructure, test-time scaling, and iterating between simulator and real system. Simulators enable **planting the answer** and working backwards, baking verifiability in. - **Scaling search with test-time compute**: trace mining, difficulty calibration ("not too easy, not too hard" — RL needs a separation between single rollouts and rollout collections), simulator refinement, rubric extraction, adversarial red-teaming. - **Reward hacking detection**: reward hacks "can kind of sneak up on you," but simple things often work — humans can identify hacks; judges struggle prospectively but hindsight reflection and building a corpus of hack examples helps. - **Validate by training**: some issues only appear once RL actually runs; fold small training runs into environment design, monitor tool-call behavior patterns. - **Human in the loop**: surface only the highest-level questions (goal, good/bad) to the human expert. ### Referenced Work - **"General agent" blog**: online loop of generate–solve–synthesize tasks gated on pass rate; reports "great uplift on popular benchmarks for tool use" (unstated specifics). - **Echo** (collaborators' work) + Prime Intellect's deep dive: agents receiving supervised learning signal from the environment itself gain a **native world model** via a likelihood model over environment tokens — enabling knowledge incorporation, not just skill refinement. ## Closing Claims - Path forward: environments as the anchor, spend compute refining them, raise abstraction for reward design "in the same way that with coding agents we're going to higher levels of abstraction." - End state: **closed loop** — models stay in guardrails, find production issues, convert them into new training tasks. - Commercial note: Prime Intellect works hands-on with startups/enterprises and is hiring. ## Qualifications & Unresolved Tensions - Brown repeatedly hedges: "we're figuring it out as we go," "we don't know how it all works." - Acknowledges judges' limits: telling a model not to reward hack doesn't prevent it in rollouts. - Admits some problems only surface during actual RL training, not up-front design. - Notes the company's own tools still require fussing with research details "today," despite the automation vision. - Transcript contains apparent transcription artifacts ("Primordial AI," "Primary and Like," "Jeph I," "Prime and Elect") — recorded as spoken; not corrected beyond obvious context. # Speaker Attribution Record ## Video **Title:** Reinforcement Learning without Verifiable Rewards — Will Brown, Prime Intellect **ID:** `01a15ca8-2db8-7447-8b73-27a60c310457` ## Speakers | Speaker | Role | Evidence | |---|---|---| | **Will Brown** | **Creator / Primary presenter** — self-identified: *"I'm Will Brown. I lead applied research at Primordial AI"* (note: title metadata says **Prime Intellect**; transcript self-introduction says **Primordial AI** — discrepancy explicit) | Entire talk | | Unidentified attendee(s) | **Audience** — no speaking lines; applause noted at end | `[applause]` | - No guests, narrators, or other identified speaking roles. ## Roles & Affiliations (as stated) - Will Brown: leads **applied research** at Primordial AI / Prime Intellect. - Company products referenced by speaker: **Primary RL training framework** (transcribed name), **Lab** training platform, environments, GPU orchestration. - Speaker says "we" throughout, indicating organizational work, not personal claims alone. ## Referenced Sources & Third-Party Claims - **"Echo"** — described as *"great work [from] some researchers that we are friends with and have been collaborating with"* — third-party research, unnamed authors. - Algorithms referenced (no attribution to specific papers): **GRPO, REINFORCE, SISPO, Jeph I / prompt optimization**. - Company blogs referenced: a blog called **"general agent"** (speaker's own organization's blog, per "blogs that we've put out"). - Broader research literature referenced generically, no specific citations. ## Promotional Segments (Flagged) - Mid-talk product overview: *"at Primary and Like we build a lot of tooling... We build the Primary RL training framework... We have a training platform called Lab..."* - Closing promotion: *"We work hands-on with startup enterprises to help them train their models. We are also hiring quite a lot. If you want to get in touch... find me after the talk."* ## Provenance Notes - No curated metadata supplied; only transcript available. - **Comment availability:** not indicated in source. - Organization name inconsistency (**Prime Intellect** in title vs. **Primordial AI** and **"Primary and Like" / "Prime and Elect"** in transcript) — likely transcript artifacts; recorded as-is, unresolved. - Content synthesizes speaker's organizational work and external literature without separating them citation-by-citation. Uh thanks all for coming to AI engineer and checking out the post training session. Um hopefully lots of fun stuff today and throughout the conference. Um I'm Will Brown. I lead applied research at Primordial AI and today I want to talk about reinforcement learning without verifiable rewards. And so many of people may have been learning about RLVR over the past uh year or so uh year and a half as this stuff has really taken off and become the main way that we think about scaling reinforcement learning. Um but often we don't actually have verifiable rewards. And so messy real world tasks often we're kind of figuring out as we go. We're having our agents run around and we kind of in hindsight maybe can like look at what they did and say like okay this is good this is bad. Um but sometimes sitting down and just like specifying okay this is the rule this is the goal is not always so straightforward. And so this is going to be uh synthesizing a lot of work we've been doing as well as from the broader research literature and some of the things we're building to kind of support extending RL into more messy real world tasks. And so recap quickly of how reinforcement learning works. I would imagine if you're in the post training session here you've probably heard a little bit about RL but for those of you at home and uh for those uh who are kind of still just kind of looking for the crash course generally we have an agent which we're going to call a model plus a harness which we place into an environment. And so an environment we're going to call a task plus a world. A world you could think of as maybe it's a docker image maybe it's a code base maybe it is a collection of uh task specific tools maybe it's some skills maybe it is a bunch of applications or browser tabs or things like this as well as a scoring rule uh verifiers or rewards whatever you want to call them uh and the agent in the environment are going to interact in a loop and at the end we'll have some reward of how well the agent did in the environment for this task. And then reinforcement learning is all about creating an advantage, and the advantage is really about taking the reward minusing some baseline, and maybe doing some scaling, and then \[clears throat\] now you have a set of rollouts from the agent in the environment that you can use to then update the policy. The policy here is just the model weights themselves, and the goal here is to take a gradient which nudges the model towards getting higher reward. And so all the RL stuff people talk about whether it's GRPO or reinforce or SISPO or any of the other new algorithms people come up with, they're all kind of in this policy gradient framework, which is just about saying, "Okay, how do I make the model do things that have higher reward?" \>\> \[clears throat\] \>\> And so at Primary and Like we build a lot of tooling to power all this at every player. We kind of go both from the we start at the compute layer and do lots of large-scale GPU orchestration. We build the Primary RL training framework, which powers all of our large-scale reinforcement learning and other algorithms running. We build environments. We have task sets and harnesses and verifiers as tools that you can mix and match to assemble complex worlds for agents to learn from real-world feedback. We have a training platform called Lab, which is anchored around environments where we do both hosted training and evaluations as well as inference. And this is to allow people to monitor their experiments and manage the training runs and iterate on their evals and deploy these models. And ultimately the models are starting generally from some open-source base model, and you're optimizing it for your task. And the goal that we're really trying to enable is more people to be able to become their own research lab, become uh take ownership over the intelligence of their own model weights and that optimize for the task that they care about with themselves as the experts steering the model, which means we need to make it way easier for people to do this. Currently, for a lot of people, it's still really hard. I think you can go like we're all here learning more about how it works because it's hard. We don't know how it all works and we're figuring it out as we go in many cases, but we've spent a lot of effort and a lot of time building stuff that hopefully makes this a bit easier for people. Um and so what's an environment? An environment is tasks, harness, and rewards. Uh but it's not just for RL. So I think a lot of people think RL when they think environment, but environments and evals are really the same thing. You can use these same objects for generating static data, which then you can use for SFT. You can do RL or you can do algorithms like on-policy distillation. You can do prompt optimization like Jeph I. You can use it as a kind of scientific test bed to iterate on your agents and your harnesses. Um and verifiable rewards are the easy case where we just kind of can check exactly was something done correctly or not. And so for math, often if you have a numerical answer, you can just parse this out of like a box in the answer from the model and check. For code, maybe you want to use test cases or a linter or something like this. For tool use, often you have some database state, which you kind of know what you what you're expecting at the end and you can just kind of like check this deterministically. And so these are kind of the easy cases where the the uh reward design problem is not so difficult. Um but most real-world tasks are not this verifiable. Uh for a lot of real agent work tasks, we're having agents do things like write reports that maybe are analyzing a bunch of documents or research. Um we're maybe asked to do things like book flights or buy things, but there isn't always like a clean best answer here. Uh and there's also notions that are fuzzier like interacting with users like handling a refund. Like what does it mean to handle this well? Um and so here the signal is less clear and there's a lot of different tricks we might want to explore and techniques we want to develop to ensure that this can be done reliably and scalably. Uh and making evals is hard because often times the benchmarks out there that we might like look at in the kind of uh new model releases, it's like a set of a few hundred tasks that a bunch of researchers spend months kind of handcrafting and talking to experts. Maybe they worked with data vendors and kind of like spent lots and lots of money kind of getting these to be very like precisely refined. And and this isn't very scalable out of the box um especially for things that are more open-ended where there's no kind of clean check for what's good or not. Um often the the the real-world situations can be unbound. You don't always know what things are going to be in the distribution. Like a lot of these cases we are figuring out the distribution as we go. And classical machine learning will tell you you can train for the distribution, but generalizing outside of the distribution is kind of an undefined problem. And then especially with RL, we have to be very careful about reward hacking. And so reward hacking is when you have a kind of loose proxy for your objective that is undefined at the boundaries. And then models, if you train with RL, they can learn to exploit this and find weaknesses where there's some path towards a climbing the the reward that doesn't actually give you what you want. And so really the goal of what we would hope all of this builds into is continual learning, which is a big buzzword that I think a lot of people like talking about in many different ways, but I'm going to use it to mean a very particular thing, which is that we want models to be deployed in relatively realistic complex, messy settings and to be able to learn as they go where they are doing things, they are making mistakes, they are then able to observe and catch these mistakes after they happen, and use this to not do the same thing again. In some cases people want to try to do this at the harness layer of the front layer. But ultimately you want a system that can evolve autonomously to be able to get better over time with humans in the loop at that right level of abstraction. And I think currently the level of abstraction for doing this is far too low for it to be practical for most people. And so this means we need new methods to be able to uh automatize as much of the difficult processes as possible. And many of these actually are automatizable. They just are kind of difficult problems to solve. And so there's a few techniques you can you can use to kind of start making progress here, but one of the goals here is to do online reinforcement learning so that you can kind of iterate on this process as you go, as well as beyond just RL. There are other things where you might want to incorporate world knowledge into the model itself, not just in terms of skill refinement. RL's great for refining skills, but less so for incorporating like dense new knowledge. And so blending these two together is also an important goal. And ultimately the what we want from this is to be able to deploy agents into production and have them improve as they go and have the these experiments be monitorable and traceable and replayable so that we can kind of treat model optimization very much as a science and make this science accessible to people who have a very wide variety of use cases they want to deploy agents for which don't all live in the training distributions of the big models. And so we want to be able to do this on top of the best and biggest open models in the world and make this accessible. And so how do you manufacture signal? There's a bunch of techniques that we've found very useful. One is grounding. And so grounding roughly means that you have some source material and in machine learning generally you want to have some notion of supervision. There's something you're learning from. And in messy situations we don't necessarily always have clean supervision, but we can get pretty reliable supervision if we kind of are careful about techniques we use. And so grounding is one where you have some source material and the ability to do an AB test of like with and without is a very useful way of creating this kind of capability gap where a model will do better if it has something in context. And this gap is something we can exploit to create signal that we can then learn from. Judges are also really useful. We're relying on the fact that LLMs are already really powerful general reasoners for many things. And if we assign the if we set these judges up in the right way, then they can spend computes to make decisions about whether an action was good or bad. We also want to be scaling search. So search in many ways is something we can apply at many different layers of the pipeline both in terms of creating tasks as well as the worlds as well as the criteria for which we want to be giving judges for answering questions about the quality of a rollout. And so for source material, um one very useful version of this, especially for this continual learning goal, is production traces themselves. And so what we found is super helpful is taking existing traces from a deployed agent and treating these as the source material where we don't necessarily know up front what the distribution of tasks is, but as an agent is deployed, you start collecting more and more examples of let's say user prompts or uh uh calls from an orchestrator agent down into a sub-agent. And this starts becoming the distribution. We don't have labels yet, but it tells us at least what we want to look for. Um and so this is one very useful category, especially for other things like uh search or for code. Uh you have doc corpora corpora corpora of documents, you have repos that are also very useful for kind of anchoring your um your uh your learning as well. That are And so uh taking these sources, these raw materials, as places to kind of search for tasks from is a very useful way of starting to kind of create this environment out of nothing. Well, it's not nothing, it's something from the real world. And because you have the real world, you want to use that, the real world, your production environment, your agent traces, as the source from which you want to learn, even if you don't have supervision yet. And so one thing you have to do here is get tasks. And so documents are actually a pretty easy version of this where you can just sample documents, you can have models generate question-answer pairs grounded in the documents, you can verify that those answer those questions are answerable with other models that are still grounded in the documents, and then the actual task at hand involves throwing away the initial search. And so you kind of get to work backwards. And so this general principle of like working backwards is starting from the solution or something that's close to the solution and have where your real task is like further upstream. This is a very useful way of kind of having a You can verify the easy problem and then learn on the hard problem. And so anything where you can move backwards like this is super useful for kind of getting supervision for free. Um in code, you can have uh you can use uh real-world PRs, the diffs, the the descriptions, the test cases, uh removing different pieces of different files to be able to have models start learning over code bases because you can kind of take something that is a completed artifact and start breaking it down into smaller pieces and then have replaying these pieces of getting to an end state that you know is reachable uh be a task that you train on. And so, this idea of wanting to know that an end state is reachable and that you can then take steps back, throw away the solution, and then learn to find it again uh can be applied more generally beyond code as well. Uh and so, we talk about world simulators broadly as the the sort of thing we might want to do in messier environments which are not just uh production uh like which are not just doc search or code. And so, a lot of the ones that we've been working on at Prime and Elect are related to things like tool use and web applications where we don't actually have full controllability of the back-end state. There are some MCP tools or CLI tools or websites or applications where we can't actually program them yet. And so, what we want to do is learn to simulate them. And so, we found that using combinations of uh universal back-end infrastructure and test time scaling and search and kind of iterating between the simulator and the real uh and the real, we can kind of ground in these production traces. This then allows us to create really high-fidelity simulators. And so, we found that these simulators are actually really great for RL because you can one, if you have production data, you can make your simulator better and better over time, but also you have full controllability over the back-end. And so, you can actually do this reverse engineering where you get to kind of plant the answer. You can start from the end and work backwards uh and so that you you have this verifiability baked into the simulator even if you don't have it in the real-world production deployment because you don't know in advance if a task was solvable at the time that you are being asked it. Um and in terms of doing this, a very useful thing is scaling judges. And so, a lot of times we will have a model that does something and it will make mistakes along the way and it's easier to tell what went wrong in hindsight. And so, the fact that you've already seen the chain of events after and you can look backwards and say, "Okay, the model made a mistake here. This thing doesn't feel quite right." Or we asked seven different models and they all kind of agree this thing is wrong. This is a very useful way of kind of spending compute to do search to then extract rubrics. These rubric questions are are are very effective at like kind of distilling down the the search into something that we can then use to more cheaply audit and kind of also ground, once we have these rubrics, as a a way of saying, "Okay, we need tasks that target these kinds of failure modes as well." Um all of this is under the umbrella of scaling search with test-time compute. And so, we can scale search for mining traces by if we have offline production traces, we can just look at them more and think about them more and have more models play with them. We can calibrate difficulty. So, RL to have the advantage gap that we mentioned needs to have a a separation between what one model will do once and what a model what a collection of rollouts will do. And so, you want tasks that are not too easy, not too hard and you want to be searching for these and iterating on generating more of them. And so, this is another area where you can spend compute to um refine the difficulty of your your task distributions, your task sets. Um for simulators, when you're building web applications or tools that need to simulate complex behavior, you can spend compute on searching these and then you can have agents refine the implementations of them and this allows for increasingly high-fidelity environments. You can do this on verification both at train time as well as offline when you're kind of creating these rubrics. You can do things like red teaming with adversarial prompt optimization to to kind of explore for backdoors. Then you can look for traces and spend compute mining these traces for understanding was this the reward a reward hacker? Was this actually kind of in the spirit of the task? And I think these things can kind of feel like reward hacking can kind of sneak up on you if you're not careful for it, but in many cases the basic simple things actually work quite well where if the reward hacks are the sorts of things where a human can look at them and be like, oh yeah, that's a reward hack. Judges are often not quite good at doing this as well. They just don't necessarily tell if you tell the model not to do this, it won't necessarily do it in the rollout, but in hindsight you can reflect on this and spend compute to kind of especially if you are collecting these over time and you are building up your corpus of examples of reward hacks, you can understand the sorts of things that go wrong and address this by kind of again spending inference compute on refining your implementation, refining your rewards. As well as validating these by training. And so we find that in many cases you can do a lot up front, but also also there are things that don't show up until you actually like start doing RL. And so part of this is folding in training experiments themselves into the process of environment design where you can you can do small runs with individual models on like one environment and and see what happens and you can understand that the behavior changes. You can have metrics that log the the types of tool calls that are being done that are like judges asking questions about the the traces to understand how behavioral patterns are changing. And all of these are very useful ways of kind of getting something from nothing and using compute as the thing that allows you to refine your understanding. And ultimately what you want is to surface the most important pieces up to the human, the the highest level of questions about what is actually going on, what is the goal so that the all this is deferring to the human for the most important pieces of like actually employing expert to say this is good, this is bad, this is what I want, this is not what I want. And uh these are all the ways that we kind of gain confidence in the environments. I know we're running a little short on time, so I wanted to break up a couple blogs that we've put out recently that are kind of demonstrating pieces of this. We have a blog called general agent which is demonstrating this for tool use, this online loop of generating, solving, and synthesizing new tasks and gating based on this pass rate which then we train on and we see a great uplift on popular benchmarks for tool use. Um additionally, beyond just RL, we found that it's quite important to think about cases where there's information in the world that RL alone will not explore. And so, there's this great work Echo from some researchers that we are friends with and have been collaborating with. And then we did our own kind of deep dive into this as well to look at what happens when you have an agent that is not just training with reinforcement learning, but is also getting supervised learning signal from the environment itself. Which then allows the model to understand things like having a native world model of the environment, understanding what to expect because it has a likelihood model of the tokens that the environment itself will generate. And these are the sorts of things that in many cases allow the model itself to kind of more adaptively navigate the world and not just like refine its skill, but like get new information into its weights over time as well. Um and so all of this is in spirit of making post-training easier, making continual learning easier, giving people the ability to create agents and not worry too much about having to to fuss with the research pieces and fine-tune all the the small details. Today we still do, but we're kind of seeing paths forward of how we start automating this more and more by like having environments as the anchor, which we can then spend compute on refining. We can kind of use compute to mine the data we have from the real world to refine the the signals where the humans are kind of just in the same way that with coding agents we're kind of going to higher levels of abstraction, we can do this with environment and reward design as well. Um and all of this is well allows us to ultimately close the loop where models are then able to stay within the guardrails we give them, they go find the issues in production, and then they turn these back into new tasks that can then be trained on for getting better in the real world. Um we work hands-on with startup enterprises to help them train their models. We are also hiring quite a lot. If you want to get in touch for either of these, find me after the talk. Thanks a bunch. \>\> \[applause\] \>\> Hey.