--- schema: 1 type: video title: >- Beyond Static Intelligence: Evaluating Continual Learning — Parth Asawa, UC Berkeley description: >- Every leaderboard you have seen was built by asking a model to do one task, wiping its memory, and asking it another. Parth Asawa's objection is that this quietly assumes learning across instances does not count. His benchmark measures what that assumption hides, using a metric called gain: run a system with state, then run the identical system reset between every single instance, and take the difference. Cumulative reward cannot show you this, because a stronger base model can post a higher total while learning less than a weaker one that genuinely improves. Building tasks that can measure learning turns out to be the hard part, and he sets three requirements. Headroom, so the task is not already solved by pretraining. Shared latent structure across instances, since standard benchmarks are deliberately independent and therefore offer nothing to improve on, which is why chaining existing benchmarks together does not work. And a learning signal in the environment, whether reward, error messages, or plain text. Continual Learning Bench 1.0 spans six domains including database exploration, where a system should need fewer SQL queries by the tenth question, after which a schema migration tests whether it can throw away stale knowledge without throwing away the useful kind. The headline result is uncomfortable: plain in context learning tops the leaderboard, beating the more elaborate context management systems on reward, on gain, and on cost. Failure modes land on either side of stability and plasticity, including a forecasting model that overpredicts, is corrected, underpredicts, is corrected again, and then jumps straight back to its original overprediction instead of splitting the difference. Speaker info: - https://x.com/pgasawa - https://www.linkedin.com/in/pgasawa/ - https://pgasawa.github.io/ Timestamps: 0:00 - How we evaluate models today 1:28 - Imagine forgetting everything after every task 2:05 - What continual learning actually means 2:42 - In context, external memory, or parametric 3:18 - The case that we are not measuring it at all 3:58 - What the existing literature does 4:37 - Why those evaluations are not enough 5:13 - Why you cannot chain existing benchmarks 5:51 - Design criterion one: headroom 7:06 - Shared structure and a learning mechanism 7:42 - Reward, and why cumulative reward misleads 8:58 - Gain: the same system with memory wiped 10:15 - Isolating learning from base capability 10:54 - The database exploration task 12:06 - Adding concept drift with a migration 13:20 - Six domains in the benchmark 13:57 - Results, and the in context learning surprise 15:12 - Failure modes on stability and plasticity 15:51 - A forecast that forgets its own correction 16:28 - A notepad that refuses to update 17:07 - Why the training stack was never built for this 17:47 - The sunk cost fallacy in continual learning 19:02 - Rethinking third party AI research 19:38 - Roadmap --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/iqloyWCGYQQ-beyond-static-intelligence-evaluating-continual-learning Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/iqloyWCGYQQ' markdown: 'https://liquidgold.cc/watch/iqloyWCGYQQ.md' llms: 'https://liquidgold.cc/watch/iqloyWCGYQQ/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T05:35:31.822Z' language: null children: [] videoId: iqloyWCGYQQ durationSeconds: 1230 publishedAt: '2026-08-12T14:00:19.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 23 endSeconds: 88 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=23' heldReason: rights - ordinal: 1 text: null startSeconds: 88 endSeconds: 99 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=88' heldReason: rights - ordinal: 2 text: null startSeconds: 99 endSeconds: 130 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=99' heldReason: rights - ordinal: 3 text: null startSeconds: 130 endSeconds: 141 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=130' heldReason: rights - ordinal: 4 text: null startSeconds: 141 endSeconds: 162 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=141' heldReason: rights - ordinal: 5 text: null startSeconds: 162 endSeconds: 173 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=162' heldReason: rights - ordinal: 6 text: null startSeconds: 216 endSeconds: 238 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=216' heldReason: rights - ordinal: 7 text: null startSeconds: 250 endSeconds: 283 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=250' heldReason: rights - ordinal: 8 text: null startSeconds: 283 endSeconds: 295 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=283' heldReason: rights - ordinal: 9 text: null startSeconds: 315 endSeconds: 326 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=315' heldReason: rights - ordinal: 10 text: null startSeconds: 358 endSeconds: 370 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=358' heldReason: rights - ordinal: 11 text: null startSeconds: 370 endSeconds: 434 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=370' heldReason: rights - ordinal: 12 text: null startSeconds: 434 endSeconds: 445 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=434' heldReason: rights - ordinal: 13 text: null startSeconds: 467 endSeconds: 478 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=467' heldReason: rights - ordinal: 14 text: null startSeconds: 557 endSeconds: 623 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=557' heldReason: rights - ordinal: 15 text: null startSeconds: 623 endSeconds: 633 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=623' heldReason: rights - ordinal: 16 text: null startSeconds: 655 endSeconds: 666 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=655' heldReason: rights - ordinal: 17 text: null startSeconds: 732 endSeconds: 743 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=732' heldReason: rights - ordinal: 18 text: null startSeconds: 743 endSeconds: 810 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=743' heldReason: rights - ordinal: 19 text: null startSeconds: 810 endSeconds: 822 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=810' heldReason: rights - ordinal: 20 text: null startSeconds: 844 endSeconds: 856 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=844' heldReason: rights - ordinal: 21 text: null startSeconds: 856 endSeconds: 920 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=856' heldReason: rights - ordinal: 22 text: null startSeconds: 931 endSeconds: 953 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=931' heldReason: rights - ordinal: 23 text: null startSeconds: 963 endSeconds: 997 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=963' heldReason: rights - ordinal: 24 text: null startSeconds: 1008 endSeconds: 1030 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=1008' heldReason: rights - ordinal: 25 text: null startSeconds: 1041 endSeconds: 1074 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=1041' heldReason: rights - ordinal: 26 text: null startSeconds: 1074 endSeconds: 1085 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=1074' heldReason: rights - ordinal: 27 text: null startSeconds: 1085 endSeconds: 1151 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=1085' heldReason: rights - ordinal: 28 text: null startSeconds: 1183 endSeconds: 1194 watch: 'https://liquidgold.cc/watch/iqloyWCGYQQ?t=1183' heldReason: rights --- # Beyond Static Intelligence: Evaluating Continual Learning — Parth Asawa, UC Berkeley Every leaderboard you have seen was built by asking a model to do one task, wiping its memory, and asking it another. Parth Asawa's objection is that this quietly assumes learning across instances does not count. His benchmark measures what that assumption hides, using a metric called gain: run a system with state, then run the identical system reset between every single instance, and take the difference. Cumulative reward cannot show you this, because a stronger base model can post a higher total while learning less than a weaker one that genuinely improves. Building tasks that can measure learning turns out to be the hard part, and he sets three requirements. Headroom, so the task is not already solved by pretraining. Shared latent structure across instances, since standard benchmarks are deliberately independent and therefore offer nothing to improve on, which is why chaining existing benchmarks together does not work. And a learning signal in the environment, whether reward, error messages, or plain text. Continual Learning Bench 1.0 spans six domains including database exploration, where a system should need fewer SQL queries by the tenth question, after which a schema migration tests whether it can throw away stale knowledge without throwing away the useful kind. The headline result is uncomfortable: plain in context learning tops the leaderboard, beating the more elaborate context management systems on reward, on gain, and on cost. Failure modes land on either side of stability and plasticity, including a forecasting model that overpredicts, is corrected, underpredicts, is corrected again, and then jumps straight back to its original overprediction instead of splitting the difference. Speaker info: - https://x.com/pgasawa - https://www.linkedin.com/in/pgasawa/ - https://pgasawa.github.io/ Timestamps: 0:00 - How we evaluate models today 1:28 - Imagine forgetting everything after every task 2:05 - What continual learning actually means 2:42 - In context, external memory, or parametric 3:18 - The case that we are not measuring it at all 3:58 - What the existing literature does 4:37 - Why those evaluations are not enough 5:13 - Why you cannot chain existing benchmarks 5:51 - Design criterion one: headroom 7:06 - Shared structure and a learning mechanism 7:42 - Reward, and why cumulative reward misleads 8:58 - Gain: the same system with memory wiped 10:15 - Isolating learning from base capability 10:54 - The database exploration task 12:06 - Adding concept drift with a migration 13:20 - Six domains in the benchmark 13:57 - Results, and the in context learning surprise 15:12 - Failure modes on stability and plasticity 15:51 - A forecast that forgets its own correction 16:28 - A notepad that refuses to update 17:07 - Why the training stack was never built for this 17:47 - The sunk cost fallacy in continual learning 19:02 - Rethinking third party AI research 19:38 - Roadmap --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/iqloyWCGYQQ-beyond-static-intelligence-evaluating-continual-learning Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/iqloyWCGYQQ · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.