--- schema: 1 type: video title: 'Scaling up Continual Learning — Ronak Malde, Trajectory' description: >- Scale on policy self distillation to trajectories with a hundred tool calls and the model collapses into hedging. The tokens it learns to favor fill up with wait, but, and maybe, until, as Ronak Malde puts it, everything just turns into maybe. He calls it the but wait problem, and it happens because the student drifts far enough on a long task that the teacher course corrects at every opportunity, leaving the model parked between two divergent distributions. The algorithm underneath is a good trick. At the frontier there is no smarter model to distill from, so you make the model its own teacher: put privileged information, a hint, in the teacher's prompt, and match the log probs of the student that never saw it. Malde scores post training methods against four properties, an online task distribution, on policy sampling, no parallel rollouts, and a per token reward, and shows SFT, RLHF, and GRPO each buying some at the cost of others. GRPO gets on policy sampling but explodes parallelism and collapses feedback into one sequence level score, which he compares to being handed 87 out of 100 on an essay and told to work out why. Self distillation gets all four, and it optimizes across the entire vocabulary at every token instead of sharpening the one that was sampled, which is why it keeps climbing past where GRPO plateaus while tokens to solve go down rather than up. The failure modes are the useful part: step level KL weighting to handle divergence, and residual guidance for hint leakage, the self distillation analogue of reward hacking, where a hint containing the answer teaches the model to state it and back fill the reasoning afterward. Speaker info: - https://x.com/rronak_ - https://www.linkedin.com/in/ronak-malde Timestamps: 0:00 - From Windsurf to Trajectory 1:05 - Benchmarks are saturating and getting expensive 1:58 - The signal we throw away every day 2:51 - Four things a training algorithm should have 3:42 - SFT, and what it got right 4:31 - DPO and RLHF 5:22 - GRPO and the Faustian bargain 6:15 - How GRPO actually works 7:04 - Scored 87 out of 100 and told to figure it out 7:52 - Distillation, then on policy distillation 8:43 - Self distillation: make the model its own teacher 10:21 - Optimizing the whole vocabulary, not the top token 11:11 - Results on short horizon tasks 12:52 - What breaks at 120B and a hundred tool calls 13:41 - The but wait problem 14:31 - Step level divergence weighting 16:11 - Hint leakage, the new reward hacking 17:51 - Residual guidance 19:35 - All four properties, finally 20:25 - What Trajectory is building 21:16 - Q&A: how continual is continual learning 22:08 - Q&A: model and harness improving together --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/zL1kLftVTlo-scaling-up-continual-learning Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/scaling-up-continual-learning-ronak-malde-trajector-f27bb798590c markdown: >- https://liquidgold.cc/channel/aidotengineer/video/scaling-up-continual-learning-ronak-malde-trajector-f27bb798590c.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T05:41:06.717Z' language: null children: [] videoId: zL1kLftVTlo durationSeconds: 1383 publishedAt: '2026-08-12T14:30:11.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 67 endSeconds: 79 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=67' heldReason: rights - ordinal: 1 text: null startSeconds: 79 endSeconds: 123 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=79' heldReason: rights - ordinal: 2 text: null startSeconds: 188 endSeconds: 222 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=188' heldReason: rights - ordinal: 3 text: null startSeconds: 233 endSeconds: 277 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=233' heldReason: rights - ordinal: 4 text: null startSeconds: 277 endSeconds: 288 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=277' heldReason: rights - ordinal: 5 text: null startSeconds: 288 endSeconds: 324 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=288' heldReason: rights - ordinal: 6 text: null startSeconds: 336 endSeconds: 381 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=336' heldReason: rights - ordinal: 7 text: null startSeconds: 480 endSeconds: 492 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=480' heldReason: rights - ordinal: 8 text: null startSeconds: 528 endSeconds: 538 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=528' heldReason: rights - ordinal: 9 text: null startSeconds: 627 endSeconds: 640 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=627' heldReason: rights - ordinal: 10 text: null startSeconds: 682 endSeconds: 780 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=682' heldReason: rights - ordinal: 11 text: null startSeconds: 780 endSeconds: 791 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=780' heldReason: rights - ordinal: 12 text: null startSeconds: 823 endSeconds: 835 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=823' heldReason: rights - ordinal: 13 text: null startSeconds: 878 endSeconds: 889 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=878' heldReason: rights - ordinal: 14 text: null startSeconds: 971 endSeconds: 982 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=971' heldReason: rights - ordinal: 15 text: null startSeconds: 982 endSeconds: 1071 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=982' heldReason: rights - ordinal: 16 text: null startSeconds: 1071 endSeconds: 1082 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=1071' heldReason: rights - ordinal: 17 text: null startSeconds: 1287 endSeconds: 1331 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=1287' heldReason: rights - ordinal: 18 text: null startSeconds: 1331 endSeconds: 1342 watch: 'https://liquidgold.cc/watch/zL1kLftVTlo?t=1331' heldReason: rights --- # Scaling up Continual Learning — Ronak Malde, Trajectory Scale on policy self distillation to trajectories with a hundred tool calls and the model collapses into hedging. The tokens it learns to favor fill up with wait, but, and maybe, until, as Ronak Malde puts it, everything just turns into maybe. He calls it the but wait problem, and it happens because the student drifts far enough on a long task that the teacher course corrects at every opportunity, leaving the model parked between two divergent distributions. The algorithm underneath is a good trick. At the frontier there is no smarter model to distill from, so you make the model its own teacher: put privileged information, a hint, in the teacher's prompt, and match the log probs of the student that never saw it. Malde scores post training methods against four properties, an online task distribution, on policy sampling, no parallel rollouts, and a per token reward, and shows SFT, RLHF, and GRPO each buying some at the cost of others. GRPO gets on policy sampling but explodes parallelism and collapses feedback into one sequence level score, which he compares to being handed 87 out of 100 on an essay and told to work out why. Self distillation gets all four, and it optimizes across the entire vocabulary at every token instead of sharpening the one that was sampled, which is why it keeps climbing past where GRPO plateaus while tokens to solve go down rather than up. The failure modes are the useful part: step level KL weighting to handle divergence, and residual guidance for hint leakage, the self distillation analogue of reward hacking, where a hint containing the answer teaches the model to state it and back fill the reasoning afterward. Speaker info: - https://x.com/rronak\_ - https://www.linkedin.com/in/ronak-malde Timestamps: 0:00 - From Windsurf to Trajectory 1:05 - Benchmarks are saturating and getting expensive 1:58 - The signal we throw away every day 2:51 - Four things a training algorithm should have 3:42 - SFT, and what it got right 4:31 - DPO and RLHF 5:22 - GRPO and the Faustian bargain 6:15 - How GRPO actually works 7:04 - Scored 87 out of 100 and told to figure it out 7:52 - Distillation, then on policy distillation 8:43 - Self distillation: make the model its own teacher 10:21 - Optimizing the whole vocabulary, not the top token 11:11 - Results on short horizon tasks 12:52 - What breaks at 120B and a hundred tool calls 13:41 - The but wait problem 14:31 - Step level divergence weighting 16:11 - Hint leakage, the new reward hacking 17:51 - Residual guidance 19:35 - All four properties, finally 20:25 - What Trajectory is building 21:16 - Q\&A: how continual is continual learning 22:08 - Q\&A: model and harness improving together --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/zL1kLftVTlo-scaling-up-continual-learning Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/scaling-up-continual-learning-ronak-malde-trajector-f27bb798590c · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.