--- schema: 1 type: video title: >- How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI description: >- Laurie Voss reran a year old benchmark and the models walked straight through its ceiling without noticing it was there. IFScale asks a model to write a business report containing a list of exact words, then counts how many actually appear. A year ago frontier models started dropping instructions somewhere around 200 to 300, which is a hard limit on how much you can put in a skills file. Voss, head of developer relations at Arize AI and a cofounder of npm, first replicated that result on the three models from the original paper still reachable by API, then pointed the same test at the current frontier. They scored 100 percent immediately. He had to raise the benchmark from 500 words to 10,000 before anything broke at all. The boundary now sits near 2,000 instructions, and for the best model closer to 5,000, roughly a tenfold gain in twelve months. The stranger finding is that the models no longer fail the same way. One simply forgets, shedding nearly half its rules by 2,000. One refuses at the API level, because a pile of random words trips its safety classifier. One spends its whole thinking budget verifying instructions and leaves no room to answer. The last writes five thousand words of the report, decides the request is stupid, says so politely, and stops, which reads like a finished answer unless you get to the end. Voss argues this moves the work: fitting rules into a small budget was a compression problem and it is gone, while knowing whether the model obeyed is a verification problem that only checking outputs solves. The whole study cost 29 dollars. Speaker info: - https://x.com/seldo - https://www.linkedin.com/in/seldo/ - https://seldo.com Timestamps: 0:00 - The 200 instruction ceiling and where it came from 1:49 - Not knowing whether it followed your rules 2:58 - How the IFScale benchmark works 4:19 - Replicating last year's result 6:12 - Pointing the same test at current models 7:07 - The headline, a tenfold jump in a year 9:01 - Four models, four different ways to fail 14:27 - What this changes in your workflow 16:29 - Caveats, and what the test does not prove 19:32 - Capacity went up, reliability did not 21:06 - Compression solved, verification remains --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/XzJD1bvXKjs-youtube-dd486a3b650838f8bba1 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/XzJD1bvXKjs' markdown: 'https://liquidgold.cc/watch/XzJD1bvXKjs.md' llms: 'https://liquidgold.cc/watch/XzJD1bvXKjs/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T04:05:19.482Z' language: null children: [] videoId: XzJD1bvXKjs durationSeconds: 1346 publishedAt: '2026-09-09T15:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 24 endSeconds: 111 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=24' heldReason: rights - ordinal: 1 text: null startSeconds: 24 endSeconds: 111 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=24' heldReason: rights - ordinal: 2 text: null startSeconds: 122 endSeconds: 188 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=122' heldReason: rights - ordinal: 3 text: null startSeconds: 122 endSeconds: 188 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=122' heldReason: rights - ordinal: 4 text: null startSeconds: 200 endSeconds: 266 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=200' heldReason: rights - ordinal: 5 text: null startSeconds: 276 endSeconds: 377 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=276' heldReason: rights - ordinal: 6 text: null startSeconds: 389 endSeconds: 433 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=389' heldReason: rights - ordinal: 7 text: null startSeconds: 445 endSeconds: 546 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=445' heldReason: rights - ordinal: 8 text: null startSeconds: 445 endSeconds: 546 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=445' heldReason: rights - ordinal: 9 text: null startSeconds: 557 endSeconds: 667 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=557' heldReason: rights - ordinal: 10 text: null startSeconds: 557 endSeconds: 667 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=557' heldReason: rights - ordinal: 11 text: null startSeconds: 677 endSeconds: 805 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=677' heldReason: rights - ordinal: 12 text: null startSeconds: 677 endSeconds: 805 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=677' heldReason: rights - ordinal: 13 text: null startSeconds: 816 endSeconds: 873 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=816' heldReason: rights - ordinal: 14 text: null startSeconds: 873 endSeconds: 885 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=873' heldReason: rights - ordinal: 15 text: null startSeconds: 885 endSeconds: 999 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=885' heldReason: rights - ordinal: 16 text: null startSeconds: 1013 endSeconds: 1178 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=1013' heldReason: rights - ordinal: 17 text: null startSeconds: 1178 endSeconds: 1188 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=1178' heldReason: rights - ordinal: 18 text: null startSeconds: 1188 endSeconds: 1268 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=1188' heldReason: rights - ordinal: 19 text: null startSeconds: 1268 endSeconds: 1278 watch: 'https://liquidgold.cc/watch/XzJD1bvXKjs?t=1268' heldReason: rights --- # How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI Laurie Voss reran a year old benchmark and the models walked straight through its ceiling without noticing it was there. IFScale asks a model to write a business report containing a list of exact words, then counts how many actually appear. A year ago frontier models started dropping instructions somewhere around 200 to 300, which is a hard limit on how much you can put in a skills file. Voss, head of developer relations at Arize AI and a cofounder of npm, first replicated that result on the three models from the original paper still reachable by API, then pointed the same test at the current frontier. They scored 100 percent immediately. He had to raise the benchmark from 500 words to 10,000 before anything broke at all. The boundary now sits near 2,000 instructions, and for the best model closer to 5,000, roughly a tenfold gain in twelve months. The stranger finding is that the models no longer fail the same way. One simply forgets, shedding nearly half its rules by 2,000. One refuses at the API level, because a pile of random words trips its safety classifier. One spends its whole thinking budget verifying instructions and leaves no room to answer. The last writes five thousand words of the report, decides the request is stupid, says so politely, and stops, which reads like a finished answer unless you get to the end. Voss argues this moves the work: fitting rules into a small budget was a compression problem and it is gone, while knowing whether the model obeyed is a verification problem that only checking outputs solves. The whole study cost 29 dollars. Speaker info: - https://x.com/seldo - https://www.linkedin.com/in/seldo/ - https://seldo.com Timestamps: 0:00 - The 200 instruction ceiling and where it came from 1:49 - Not knowing whether it followed your rules 2:58 - How the IFScale benchmark works 4:19 - Replicating last year's result 6:12 - Pointing the same test at current models 7:07 - The headline, a tenfold jump in a year 9:01 - Four models, four different ways to fail 14:27 - What this changes in your workflow 16:29 - Caveats, and what the test does not prove 19:32 - Capacity went up, reliability did not 21:06 - Compression solved, verification remains --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/XzJD1bvXKjs-youtube-dd486a3b650838f8bba1 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/XzJD1bvXKjs · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.