Talk by Ronak Malde (Trajectory) on scaling continual learning via on-policy self-distillation. The model is made its own teacher by placing a privileged hint in the teacher's prompt while matching the log probs of the student that never saw it, satisfying four properties: online task distribution, on-policy sampling, no parallel rollouts, and per-token reward. Malde scores SFT, DPO/RLHF, and GRPO against these, arguing GRPO's sequence-level score and parallelism costs cause plateaus. Failure modes at scale (~120B models, ~100 tool-call trajectories) include the "but wait" problem, where teacher course-correction drives hedging; addressed via step-level divergence weighting and residual guidance for hint leakage, the self-distillation analogue of reward hacking. Timestamps cover benchmarks saturation, the four properties, method mechanics, short-horizon results, scaling failures, and Q&A. Topic inference beyond the supplied title and description is qualified accordingly. # Channel Presentation Observations: "Scaling up Continual Learning — Ronak Malde, Trajectory" ## Observable Context & Format - **Conference/talk format**: The video is a recorded technical presentation with a live audience, evidenced by the opening ("Hey everyone. Uh, thanks for coming by over here"), closing applause, and an audience Q&A session at the end. - **Part of a multi-talk event**: The speaker references a preceding related talk: "Hope you enjoyed the chat by part on uh, how to measure continual learning. Now I'm here to talk about how we scale it up." ## Speaker Identity (Self-Described) - **Speaker**: Ronak Malde, identified in the title as affiliated with "Trajectory." - **Background claims (unverified)**: "I went to this company called WinSurf where I was growing the research team... we trained this model called sui 1 that ended up leading to the two billion acquisition at deep mind and then I ended up giving up all the acquisition money to start trajectory." ## Intended Audience - **ML researchers and practitioners familiar with RL post-training**: The speaker assumes familiarity with GRPO, DPO/RLHF, SFT, and distillation — "hopefully you guys are all familiar with how gpo works high level" — and references shared experience: "if you guys have ever trained RL models is the models like to think a lot." - **Attendees tracking AI scaling trends**: References to "the massive amount of money that the labs are pouring into scaling up RL environments and data" and named figures (Ilia, Karpathy, Schol, Demis, Satya, Daresh) presuppose an audience current with AI industry discourse. ## Claimed Scope of the Company/Channel Subject - **Company mission (explicit claim)**: "we're building the platform for continual learning" and "building the platform that turns every interaction into model improvement, harness improvement, and the entire agentic loop just getting smarter over time." - **Product claims**: A platform that "take[s] in agent traces data from production," optimizes them in a "self-served loop," deploys them as a "continually learning system," with a "control plane that goes over all of that." - **Team credentials claim**: "Our team is super awesome. We are from DeepMind, Meta Super Intelligence, OpenAI." - **Customer traction claim**: "We've given early access to a lot of companies. So, Harvey, Dakagon, Rogo, and they're super excited about what we're doing." ## Technical Content Demonstrated - **Algorithm walkthrough**: Detailed explanation of a novel method, "on policy self-distillation" (OPSD), contrasted against SFT, DPO/RLHF, and GRPO along four axes: task distribution, on-policy sampling, parallelism, and reward granularity. - **Scaling failure modes and fixes**: Describes problems at 120B–1T parameter scale ("the buttweight problem," "hint leakage") and proposed solutions (step-level KL divergence weighting, LLM hint filtering, residual guidance). - **Result claims**: "we're actually able to scale this up to a 12b model on mercur apex agents which often requires 100 and or plus tool calls... it is a really powerful algorithm that has even surpassed RL as well" — presented without accompanying data in the transcript. - **Open-source ecosystem awareness**: Mentions compatibility with "open claw rl" for chatbot learning "with unstructured data." ## Tone & Presentation Style - **Promotional-but-candid**: Blends pitch elements (team pedigree, customer names, "let me know, keep in touch") with frank limitations — "I wouldn't say we're anywhere close to the end solution" and "It's not necessarily the algorithm to solve continual learning." - **Forward-looking framing**: Positions the work within an industry narrative — "the most exciting unlock that's going to happen in 2026 2027." ## Claims vs. Demonstrated Patterns - **Claims (creator-stated, unverified)**: acquisition history, model scale (120B–1T), customer names, team pedigree, benchmark results (LiveCodebench, GRPO "saturates around sonnet level"). - **Demonstrated**: Deep technical fluency in RL/distillation methods; willingness to disclose failure modes; audience engagement via substantive Q&A (latency, harness improvement). # Delivery Observations: "Scaling up Continual Learning" — Ronak Malde ## Tone & Overall Register - **Conversational, informal conference-talk style.** Opens casually: *"All right. Hey everyone. Uh, thanks for coming by over here."* - **High enthusiasm, repeatedly signaled explicitly.** Phrases like *"this is really exciting,"* *"really cool part,"* and *"super super awesome"* recur throughout. - **Frequent filler and self-correction:** heavy use of *"uh"* and *"um"*, plus mid-sentence restarts like *"called self-distillation policy optimization or sorry first we'll we'll talk through why GRPO actually is still not enough."* ## Vocabulary & Terminology Habits - **Dense ML jargon used without definition for the in-group audience:** *"on policy,"* *"rollouts,"* *"log probs,"* *"KL divergence,"* *"catastrophic forgetting,"* *"advantage,"* *"reward hacking."* - **Coined/branded shorthand:** introduces **"OPSD"** and **"on policy self-distillation"** as named concepts, and *"the buttweight problem"* — an informal self-coined label: *"the first part is actually really funny and what I call it the buttweight problem."* - **Self-aware correction of vocabulary:** *"there's this really interesting wordle in the the uh bottom left that you can see or wordcloud"* — corrects "wordle" to "wordcloud." ## Teaching Habits & Structural Patterns - **Checklist-of-criteria framing.** Repeatedly organizes material around four criteria (task distribution, on-policy sampling, parallelism, reward granularity), walking through each across SFT → DPO/RLHF → GRPO → OPSD. - **Step-by-step build-up teaching pattern.** Explains distillation in layers: *"So this is normal distillation. Now there's a new innovation called on policy distillation"* → *"Well, here's the final algorithm."* - **Analogy-based explanation, a clear recurring habit:** - *"it's almost like you're drinking through a straw in order to get the reward"* - *"imagine you were trying to write an essay and your teacher just gave you a score of 87 out of 100"* - *"this is actually how humans learn, right? We're continuously updating in the real world"* - **Direct audience address and assumed familiarity:** *"hopefully you guys are all familiar with how gpo works high level"*; *"if you guys have ever trained RL models is the models like to think a lot right."* - **Rhetorical question transitions:** *"So what's the problem why aren't we here yet"*; *"Now we might be saying uh you know opsd is is awesome and and we've solved this problem."* ## Recurring Phrases - **"really exciting" / "really cool"** — appears many times, e.g., *"this is a really exciting part of self-distillation,"* *"and then finally, and this is really cool."* - **"right?"** as a frequent tag seeking agreement: *"and this is actually how humans learn, right?"*, *"which is extremely powerful right."* - **"So" as a pivot word** beginning most new sections and slides. - **Hedged intensifiers:** *"a little bit,"* *"kind of,"* e.g., *"GRPO which is kind of the mode that we're in right now,"* *"things start to break apart a little bit."* ## Narrative Arc Pattern - **Problem → solution → new problem cadence.** Consistently presents a fix, then undercuts it: *"So we might be thinking okay continual learning is solved we can all go home and and be super happy. Uh the thing is this works really well for small models."* - Names failure modes vividly: *"hint leakage"* introduced as the OPSD analogue of reward hacking — a **parallel-mapping habit** (*"there is an equivalent for OPSD as well"*). ## Q&A Delivery - **Affirming before answering:** opens replies with *"Yeah, that's a great question"* and *"Totally."* - **Comfortable expressing uncertainty/openness:** *"I wouldn't say we're anywhere close to the the end solution"*; *"it's completely unexplored territory."* ## Uncertainty Notes - Fillers and disfluencies are pervasive enough to treat informal spoken register as a **stable pattern**, not isolated moments. - The four-criteria framework appears at least three times, confirming it as a deliberate structural device rather than coincidence. - Audience reactions (applause, laughter) are not visible in the transcript beyond the closing *">> [applause]"*, so delivery feedback loops mid-talk cannot be assessed. # Position Record: Scaling up Continual Learning — Ronak Malde, Trajectory ## Speaker **Ronak Malde** (identified by title metadata), founder of Trajectory; previously grew the research team at Windsurf, where work on a model ("sui 1," as transcribed) he says led to a DeepMind acquisition. ## Core Thesis - AI progress has relied on rapidly scaled benchmarks, but these are becoming **more time-consuming, expensive, and disconnected from real-world usage**. - The **"trillion token problem"**: hundreds of trillions of inference tokens per day generate rich data on model failures and successes — signal that should be captured and trained on. - Continual learning (learning from real-world interaction, as humans do) is the next major unlock; Malde cites broad field interest (names as transcribed: Ilia, Carpathi [sic], Schol, Demis, Satya, Daresh) as context. ## Diagnosis: Why Current Algorithms Fall Short Malde identifies four recurring axes of evaluation: 1. **Task distribution mismatch** — benchmarks not tied to reality. 2. **Off-policy sampling** — methods often don't sample truly on policy. 3. **Parallelism / rollouts** — multiple rollouts require heavy infrastructure and one-to-one environment copies, introducing bias. 4. **Coarse rewards** — collapsing everything into a single scalar when real-world signal is noisy and per-token. ### His historical walkthrough - **SFT** (instruction tuning, GPT-3.5/ChatGPT era): parallelism solved (one example per rollout), per-token reward; but off-policy and benchmark/dataset task distribution. - **DPO / RLHF**: online task distributions gained; sampling from a model but still off-policy; lost SFT's simple infrastructure (pairs); reward regressed to sequence-level. - **GRPO** (current mode): a "Faustian bargain" — gained on-policy rollouts (powerful, mitigates catastrophic forgetting) but has off-policy task distributions, exploded parallelism, and sequence-level rewards. ## Proposed Algorithm: On-Policy Self-Distillation (OPSD) Stated progression: 1. **Normal distillation**: fit student log probs to a smarter teacher on a fixed (off-policy) dataset. 2. **On-policy distillation**: swap fixed data for the student's own rollout. 3. **Self-distillation**: no smarter teacher exists at the frontier, so give the student **"privileged information" (a hint)** in the teacher's prompt, then match the student's log probs (without hint) to the teacher (with hint). Simplest hint form: a golden solution example. ### Claimed properties (all four axes solved simultaneously) - **Online task distribution** — no curated benchmarks needed. - **On-policy sampling**. - **Single-example parallelism** — no group of eight rollouts; removes environment bottleneck. - **Dense per-token reward** — matches log probs for the entire vocabulary (e.g., 65k tokens), shifting whole distributions rather than just sharpening the top token as RL does. ### Reported results (short horizon) - On **LiveCodeBench**: GRPO "saturates around Sonnet-level performance"; OPSD reaches "brand new territory of results." - Token efficiency: unlike RL models that benefit from longer reasoning, OPSD **collapses tokens needed** for hard tasks. - Usable plug-and-play now with open source projects (e.g., OpenClaw RL) for chatbots learning from unstructured behavior data. ## Scaling Limitations (Trajectory's findings) At **120B–500B–1T parameter models** with 50–100 tool calls, things "break apart": - High eval accuracy variance and run-to-run variance. - **Tool call errors** — model deviates from its trained format. ### Problem 1: The "buttweight" problem - On long tasks, student diverges; the teacher course-corrects at every opportunity, repeatedly boosting hedging tokens ("wait," "but," "maybe"). - Word cloud shows hedging words emerging as steps progress; can end in a local suboptimum where "everything just turns into maybe," the model averaging two divergent distributions. **Proposed fix — step-level divergence weighting:** - Track KL divergence between student and teacher over time; use it as a **multiplicative token weight per step**, not a KL penalty. - Effects: in-distribution trajectories weighted 1; divergent trajectories train only the first step (W1) before moving on; and late re-convergence after a mid-trajectory divergence can be mildly weighted in. ### Problem 2: Hint leakage (OPSD's analogue of reward hacking) - If the student couldn't plausibly know the hint, it short-circuits reasoning. Example: hint reveals "last three digits are zeros"; rollout says "I know the solution is 000" and back-fills reasoning — behavior that "is not going to occur whatsoever in the real world." **Proposed mitigations:** - **LLM hint filtering** (trivial, works decently): translate a leaky hint (e.g., "their SSO token expired") into what's plausibly knowable (e.g., "check the logs"). - **Residual guidance** (more satisfying): use a partial hint teacher and full-hint teacher, take a **linear combination** of log probs to estimate hint strength / out-of-distribution degree, avoiding shifting the model into unknown territory. ### Scaling claim - With these fixes combined, scaled to a **12B model on Mercur Apex agents** (tasks requiring 100+ tool calls); claims it "has even surpassed RL." ## Qualified Conclusions - OPSD is "**not necessarily the algorithm** to solve continual learning" but a major step: on-policy, online task distribution, single parallelism, per-token dense reward. - Vision: software that "gets smarter every single time it's used" — predicted as the key unlock in **2026–2027**. ## Company Pitch - Trajectory builds a platform turning production agent traces into model improvement, harness improvement, and a continually improving agentic loop, with a control plane. - Team claimed from DeepMind, Meta Super Intelligence, OpenAI; early access given to Harvey, Dakagon, Rogo (as transcribed). ## Q&A Positions - **On latency/real-time learning**: Malde says the field is in "**pseudo continual learning**" — still batch updates, offline processing, model re-upload. Remaining problems are both infrastructural (merging ~10,000 concurrent production rollouts) and algorithmic; "I wouldn't say we're anywhere close to the end solution." - **On harness improvement**: Calls model–harness interplay during joint updating "one of the most underexplored and most exciting questions"; being explored with current customers; "completely unexplored territory." ## Preserved Tensions / Qualifications - OPSD works "incredibly well" for short-horizon/small-model tasks, but "academic papers kind of end" there — limitations only appear when scaled. - The teacher-with-hint is only "a little bit smarter"; hint design is delicate and leakage-prone — the algorithm's central vulnerability alongside divergence collapse. - Speaker explicitly frames claims about trajectory of field (2026–2027 unlock) as his own prediction, not established fact. # Speaker & Provenance Observation Record ## Video **Title:** Scaling up Continual Learning — Ronak Malde, Trajectory **Video ID:** `01a15ca7-fbd1-7fc1-8745-f27bb798590c` ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Ronak Malde** | **Creator / Presenter** (solo talk with Q&A) | Introduces himself; founder of Trajectory; formerly grew research team at WinSurf | | **Audience questioners (unnamed, 2)** | **Guests / Q&A participants** | Two questions after applause: one on latency/learning time, one on harness effectiveness | ## Presenter Self-Described Provenance - Formerly at **WinSurf** ("growing the research team"); claims the **sui 1** model there led to a **"two billion acquisition at DeepMind"** (as stated by speaker). - Says he "gave up all the acquisition money" to found **Trajectory**, building a continual learning platform. - Team claimed to be from **DeepMind, Meta Super Intelligence, OpenAI** (speaker's own claim). ## Referenced Third-Party Claims & Sources (as cited by speaker, not verified) - Named individuals allegedly discussing continual learning: **Ilia, Karpathy, Schol (unclear transcript spelling), Demis, Satya**; a "great video by Daresh a couple days ago." - Models/methods discussed: **GPT 3.5, ChatGPT, SFT, DPO, RLHF, GRPO, OPSD (on-policy self-distillation)**. - Benchmarks/claims: **LiveCodeBench** ("GRPO saturates around Sonnet level performance" — speaker's claim). - Open-source project cited: **OpenClaw RL**. ## Company / Promotional Segments ⚠️ - **Promotional content flagged:** opening bio (acquisition story), the final segment describing **Trajectory's platform** ("turns every interaction into model improvement… control plane"), team pedigree, and named early-access customers **Harvey, Dakagon, Rogo** ("super excited"), plus closing call to action ("let me know, keep in touch"). - Self-reported research scaling claims: OPSD scaled to **120B / 500B / "one trillion" parameter models**; "12B model on Mercur Apex agents" — all speaker claims, unverified. ## Availability & Limitations - No curated metadata supplied; identity claims rest solely on the transcript. - Some names likely garbled by transcription (e.g., "Ilia Carpathi Schol," "Daresh," "sui 1," "WinSurf," "Mercur") — uncertain, flagged as-is. - **Comment availability: not indicated** in provided material. - Audience questions are partially inaudible/elided in transcript. All right. Hey everyone. Uh, thanks for coming by over here. Hope you enjoyed the chat by part on uh, how to measure continual learning. Now I'm here to talk about how we scale it up. So a bit of background about me. I uh went to this company called WinSurf where I was growing the research team over there. Uh we trained this model called sui 1 that ended up leading to the two billion acquisition at deep mind and then I ended up giving up all the acquisition money to start trajectory where we're building the platform for continual learning. So uh right to dive in I think it's useful to talk a little bit about what AI progress uh has looked like for the past couple years. So we've been in this mode where we've been rapidly scaling up benchmarks that have saturated within first years and then months and that has continued on for the past couple years and and will continue to grow as well. The problem is it's becoming more time consuming and more expensive. Uh we're seeing domains where it takes 4 hours, 6 hours, 24 hours or even several days in order to scale up benchmarks. And obviously you guys have seen the massive amount of money that the labs are pouring into scaling up RL environments and data. Now the thing is uh we are left with a bunch of benchmarks that are getting more time consuming uh more and more expensive and and perhaps more concerningly uh they're not tied to real world use cases where people are using AI. Now the uh trillion uh token problem here is that we are actually spending hundreds of trillions of tokens every single day on inference uh and we're generating great amounts of data on how models in the real world are are failing, how they're doing well. And that should be signal that we should be capturing and training on. Uh and this is actually how humans learn, right? We're continuously updating in the real world and getting smarter every single day. And uh we're seeing now the entire field talking about continual learning from AI experts like Ilia Carpathi Schol and Demis to also many industry leaders as well. Satya talking about how it helps companies and there was actually a great video by Daresh a couple days ago talking a lot about the intricacies of scaling up continual learning and obviously the goal is we started with more data training on internet skill pre-training scaling up to benchmarks but the real unlock moving forward will be continual learning so what's the problem why aren't we here yet the main thing is uh there's several problems right now with our current algorithms so first is we have a task distribution mismatch. We're scaling up these benchmarks which might not even be tied to reality of what people are training on. And second, uh oftent times some of these methods aren't even sampling truly on policy with what what's happening online. Third is uh we have a paradigm right now where we're sending out multiple rollouts and this requires huge amounts of infrastructure to make sure our environments are onetoone copies of the real world. uh and it ends up being a bias that we're adding to our training paradigm. And then finally, third is we are shoving every single reward into one scaler in order to train on when the real world is messy. It's noisy and it has rich amounts of data uh that should be per token signal. So we have all of these kind of broken criteria uh and we're training on it. And I want to take a little bit of a step through memory lane to to see how we've been dealing with algorithms in the past. Um so the first part you know we started with SFT these were the the days of instruction fine-tuning GPT 3.5 chat GBT and uh if you take a look at these four criteria I mean we had parallelism solved back then it was just one roll or one uh use case or example that we needed to train on so that was a solved problem and then the reward for SFT is per token which is great but we are not sampling on policy and the task distribution is just some sort of benchmark that we've cured or some sort of data set. Then we moved on to the DPO, RLHF age. Uh this is when Chad GBT really started taking off and we finally got online task distributions that we were able to train on. Uh but sampling it was a little bit better because we were actually sampling from a model, but it was still off policy. But then we ended up losing some of the uh the key infrastructure, easy infrastructure that we had with SFT. Now we suddenly had pairs that we were training on. Uh and then the reward went back to sequence level. Then we moved on to GRPO which is kind of the mode that we're in right now. Uh we basically took a Fouian bargain and wanted to max on on policy rollouts which is extremely powerful right we now have models that are capable of amazing things uh because they are on policy and able to grow and and not have all of the catastrophic forgetting problems that we've had with previous methods. But on the other hand, we're working with off policy task distributions. Uh our parallelism is now exploded, meaning that we need really robust environment infrastructure. And then finally, for rewards, uh we're back to this paradigm of training on the entire sequence. So can we get to a world where all of these are true where we have an online task distribution where sampling on policy we have one parallelism so we don't need any of this crazy infrastructure and then finally our reward is actually token level well this is what I'm excited to share and and see how we can scale this up so first let's just go through what our best algorithm of post training is today hopefully you guys are all familiar with how gpo works high level. Uh you start off with some sort of task as an input. You do a bunch of parallel rollouts. So we'll call these 01 through 04. And then you have some sort of endstate reward. So you classify these, let's say, as a couple numbers. And then GRPO works on advantage. The idea that you have some sort of mean that you're calculating over and you're trying to shift the distribution of the model to be uh better on the ones that you're better on uh and then obviously move away from the ones that were worse than the mean. So this is how GRPO works. Now let's take a look at a new continual learning algorithm called self-distillation policy optimization or sorry first we'll we'll talk through why GRPO actually is uh still not enough. So um the first part is uh if you remember the tasks so these benchmarks are obviously very different. Uh it requires a bunch of parallelism and then it's almost like you're drinking through a straw in order to get the reward. The way to think about it is imagine you were trying to write an essay and your teacher just gave you a score of 87 out of 100. You would have to run through so many different examples to get to the idea of what a good essay is and it's very simple and efficient. And so these are the fundamental problems with RL. Now uh we're we're going to go through a new algorithm uh called on policy self-distillation. So you guys are probably familiar with distillation in general. Uh there's been obviously a lot of talk about mythos and and blocking that from happening for a lot of open source models. Uh the high level here is that you start with some sort of data set. Uh it's usually uh off policy and not actually tied to reality but some sort of fixed data set. And what you have is a student model and then a smarter teacher model. So usually a bigger model uh and you're trying to basically fit the log probs of the student to the teacher and both of these are passed in with the same data. All right. So this is normal distillation. Now there's a new innovation called on policy distillation. The only thing we're doing is actually swapping out the fixed data for instead a roll out of what the student would have rolled out. All right. So we take a a now roll out of the student that is our trajectory and then we're trying to fit the log props again of a smarter teacher model to the student. Great. Now the problem is when we're trying to push the frontier we don't magically have some smarter model, right? uh like a teacher. And so now what do we do if we're we're already at the smartest model? Well, here's the final algorithm. Uh and this is where the self distillation part comes in. If you basically take the the student model and give it some sort of what we call privileged information, a hint about the world and put that into the prompt. Well, suddenly that student is a little bit smarter. And that's essentially the key idea of on policy self-distillation. You take what's called this hint, put it into the beginning of the prompt, and now you match the log props of the student without that hint to the teacher with that hint. And this is an extremely powerful algorithm. To visualize it a little bit more, let's say you have this student prompt, right? Uh find the derivative of this function. What a hint would be is if you had some sort of environment information or or some sort of guidance on how you should actually solve the problem. The simplest form of this is is literally just a example of a golden solution. You put that into the teacher prompt and then say here is guidance on how you would solve the problem. And you can imagine now that solving that problem from the teacher's perspective becomes a little bit easier and we're we're trying to shift to those log problems had it known the answer in the first place. Um and so by doing this we've solved several key problems that existed with RL. Uh so if you remember uh the task distribution now we can suddenly take something that is truly from online without needing to have sort of benchmarks that are created. Uh the second part is we're still on policy sampling which is great. Uh but now there's no parallel rollouts. We don't need a group of eight in order to roll out but just from a single example we're able to get information. So that takes away the environment bottleneck and all of these other infrastructures. And then finally the most exciting part is we're matching every single log problem of every token. So there's massively rich feedback about what this algorithm is doing. Uh to see it a little more in action. And this is a really exciting part of self-distillation is it's actually not just the top token that you sampled uh that you're actually making better, but instead the entire vocabulary. So for every single token, there's a vocabulary of let's say 65k tokens that you're optimizing over. And let's say the model generated some sort of uh for loop or range in Python. And essentially what we're doing is saying, hey, the teacher for some of these areas where it's blue, it wasn't the top sample token of the student, but instead we're pushing that distribution to sample a brand new token. And this is really exciting because we're not just taking now a distribution like RL and slightly sharpening it, but we're instead actually shifting entire distributions. So this is a really exciting part about opsd. Now we might be saying uh you know opsd is is awesome and and we've solved this problem. So I mean for short horizon tasks it works incredibly well. If you take a look at live codebench uh actually we found that gpo saturates around sonnet level performance and and doesn't really push the frontier but because we're actually shifting distributions we're able to get to brand new territory of results with a lot of data sets. And then the really cool part is um with RL a fundamental limitation and if you guys have ever trained RL models is the models like to think a lot right the more tokens that you expend it's it's just going to do better but with opsd you don't have that problem and so the actual tokens to solve some of these really difficult challenges actually collapses which is really exciting for token efficiency and uh for some of these short horizon tasks you can actually plugandplay opsd right now with several open source projects So uh open claw is a really good example of this open claw rl uh and you can actually use it to do simple chat bots learn from your behaviors with unstructured data uh which is super super awesome. All right so we might be thinking okay continual learning is solved we can all go home and and be super happy. Uh the thing is this works really well for small models short horizon tasks like something like a chatbot. Uh but this is where academic papers kind of end uh and where you really need to scale things up to start to to see the limitations. Uh so at trajectory we've been scaling up this algorithm to 120bs to 500 bs to one trillion parameter models. And as soon as you get to the 120B range with not just one or two tool calls but 50 or 100 well things start to break apart a little bit. So uh first of all eval accuracy is all over the place right the range is going really high uh run-to-run variance is extremely high and then also we start to see a lot of tool call errors uh the model is not behaving accordingly to the format that it was trained on in the first place with the instruction fine-tuning all right so what are the problems that we started to see with this algorithm as we were scaling it up so the first part is actually really funny and what I call it the buttweight problem. So on shorter tasks, you're fitting to a distribution. Everything's great. But when you move on to longer tasks, what you get is is basically this student model. It's going off and doing whatever it thinks is on policy. And then at some point, because you're so divergent in a long task, the teacher is going to try to course correct every single time it gets a chance. And so what you end up with is the teacher model just uh continuously trying to improve this token of wait or uh maybe or some of these kind of hedging words. And there's this really interesting wordle in the the uh bottom left that you can see or wordcloud uh that as steps go on with opsd and you really scale it up uh you start to see some of these words like wait and then but uh start to appear. And then you actually end up in this really interesting local suboptimal position where everything just turns into maybe. And on the right here you can kind of see some visualization of this. As the task goes on you get two different distributions that are really divergent. And what you end up with is the model trying to be in the middle of both of those. Uh which is obviously really suboptimal. All right. So there's a couple ways to solve this. Uh one potential solution is to define step level divergence. So uh in a tool calling trajectory, let's say you have a 100 tool calls going on. Well, what we can do is start to look at the KL divergence of the student model and the teacher model as time goes on. And now we can use this as a waiting factor. So not just like normal KL where we use that as a KL penalty, but instead we're actually multiplying the token weight uh of every single step based on this divergence property. And now the cool part of this uh is that so on the left side we have a normal trajectory that is like pretty in distribution and everything's fine, right? It's everything is a weight of one. On the second part, we have a trajectory that diverges pretty heavily. And so in here, we're only going to modify W1 as the first step. Uh and and so that we can train that, get that right, and then move on, which is awesome. And then finally, on the third part, and this is really cool, by having independent waiting for every step is we can actually have a scenario where uh the model might go off tracks. We don't want to heavily weight that in, but then later on in the trajectory, it might get back on track again. and and we're fine with that and we'll mildly shift the distribution. So this is one way that we've been able to overcome this for a long horizon tool calling. The second part and this is actually really nefarious with uh ops. So with RL the number one problem that people face is reward hacking as you guys are are all aware of uh and that's a game that continuously RL researchers have had to play. Well there is an equivalent for OPSD as well and that is hint leakage. The way you can think about it is we're we're taking this hint, right? Putting it in the the beginning of a prompt. Well, if the student had no way of knowing what that hint would be, uh then you're going to end up with some weird scenarios and and kind of skipping some steps along the way. So, the way this manifests u is let's take this example here where you're trying to find the last three digits of of this formula. uh and a normal hint might be uh actually giving the correct steps and then maybe a final answer and saying, "Hey, actually hint, the last three digits are all zeros." Well, you can see on the right here that when you actually roll out the model, uh you end up with something that's says, "Oh, actually, I know what the solution is. It's 00 0. So, let me go back and put that into my reasoning trace and then figure out what's going on." Well, you can imagine that this is not going to occur whatsoever in the real world. And so we've ended up in this really strange position. Uh and so there's a lot of care that you have to put into how you design these hints and making sure there's not leakage of information. So uh there's one kind of trivial solution to this that you can imagine and this is just literally using an LLM to filter out these hints. So an example of this is uh you know let's say a user can't log in. you have some sort of problem like this uh and there might be some sort of information like exactly the solution right you'll find their SSO token that is expired and you can have an LLM basically translate that into what is something reasonable that they should have known and that's the process of looking through the logs but not actually giving it the solution that it would shortcut some of its vital reasoning. So this is one trivial uh solution and it works decently well but there's some more satisfying algorithmic approaches too. One of these is called residual guidance. The general idea is in a hint uh most of the time you need to actually get through the entire hint in order to get the full information. So what if you let's say cut it in half then you have a partial hint and then you basically say okay this is the partial teacher we're able to get the log props of a slightly smart teacher and then we have the full teacher as well and that's with the full hint. Now what we can do is actually take the linear combination of both of these and this gives us a good idea of how strong the hint is and how out of distribution it is for the original model. And what you end up with here is so on the left you have normal OPD uh where you might be entirely shifting distributions and there's almost no overlap between your model and the original solution. uh to this kind of cool world where uh let's say half the hint is actually quite close to your distribution but the full hint is very off and you take a linear combination and so then you're not shifting the model into unknown territory. So these are just some of the solutions to uh a lot of the challenges that we've had with scaling up opsd. Uh but it is a very very powerful algorithm and uh with a lot of these combinations together we're actually able to scale this up to uh a 12b model on mercur apex agents uh which often requires 100 and or plus tool calls in order to achieve and it's a really powerful algorithm that has even surpassed RL as well. So now we've finally arrived at a algorithm. It's not necessarily the algorithm to solve continual learning, but uh definitely one that is a huge step forward that keeps the on policy nature that makes RL so powerful. But then it also has uh finally the task distribution that is online. Uh it has parallelism uh that is singular. So we don't need all of this infrastructure. And then finally it is per token dense reward. So the uh the really exciting part for us and and what we're really focused on now trajectory is uh this just gives you one taste of the entire continual learning loop and there is a really exciting world that is about to come where software in general just gets smarter every single time it's used and that is the most exciting unlock that's going to happen in 2026 2027 and as we scale up. So a little bit about trajectory. We are uh building the platform that turns every interaction into model improvement, harness improvement, uh and the entire agentic loop just getting smarter over time. Um and we're we're building this platform where we take in agent traces data from production. We're able to optimize that uh as a self-s served loop and then deploy that uh as a continually learning system. And uh we have a control plane that goes over all of that. Um, so very quickly, our team is super awesome. We're from DeepMind, Meta Super Intelligence, OpenAI, uh, and a lot of great product builders as well. Um, and we we've given early access to a lot of companies. So, Harvey, Dakagon, Rogo, um, and they're super excited about what we're doing. Uh, if you're interested in any of the research that we're doing, uh, or any of the product that we're building, uh, definitely let me know. Keep in touch and, uh, happy to answer any questions. \>\> \[applause\] \>\> Is it in the sense of like very little latency so that you can learn time or is this something that \>\> Yeah, that's a great question. Uh so what I would say is as a research community right now we're in this zone of what I call pseudo continual learning. uh where there's some still level of like batch updates offline uh and then re-uploading the model. I think it's partly an infrastructure question. It's partly still an algorithmic question as well of how do you truly get when you have 10,000 rollouts going out in a product merging those together the infrastructure to pull all of those together. So those are some of the problems that we are solving at trajectory. Uh but I wouldn't say we're anywhere close to the the end solution. \>\> Question about hardness. studies in terms of effectiveness. \>\> Totally. I I think that's actually one of the most underexplored and most exciting questions is not only just harness improvement, right? And I think there's some literature out there now starting to explore that. But the really exciting part is how does the model and the harness interplay with each other as you're both updating them. That's some of the stuff that we're now exploring with our current customers and and really doing those things online. Uh, but it's completely unexplored territory and there's some really exciting innovations to be made there as well. \>\> Cool. All right. Thanks so much, guys.