--- schema: 1 type: video title: 'Stop Evaluating Models Like It''s the 50s - Alejandro Vidal, Mindmakers' description: >- Psychologists spent the last century learning how to measure something invisible and uncooperative: a human mind. AI evaluation, meanwhile, still scores like it is 1950. Count the right answers, treat every question as equal, trust the percentage (this is Classical Test Theory). We are sitting on decades of measurement theory built for exactly this problem, and we forgot to use it. Borrow it and the picture changes. Item Response Theory (or IRT, the math behind the SAT and the GRE) models every item on top of a shared scale with real error bars. That tells you which of your test items are pure noise, which are optimal, and where the knowledge gaps and unexpected behaviours are. Adaptive testing then measures the same ability with a fraction of the questions, which means private, rotating benchmarks that resist contamination instead of saturating in a month (tinyBenchmarks already hinted you can shrink a benchmark with IRT). It goes further than scoring. The statistical properties of how a model fits the test reveal something a single number never could: data leakage, the moment an agent has quietly seen the answers before. The same machinery that catches a cheating student catches a contaminated benchmark. And instead of one flat score, you get a shape: where the jagged frontier actually is, which abilities are solid and which are luck, so you know which direction to push next. You will leave this talk with a way to build evals that are cheaper, harder to game, and that tell you what your model actually learned instead of how lucky it got. This is not about handing human tests to a model. It is about borrowing a century of how to measure a mind that does not want to be measured. Speakers: - Alejandro Vidal (Mindmakers): Alex Vidal is the founder of Mindmakers, a psychologist and computer scientist who teaches humans to use AI and teaches AIs to teach humans, building adaptive learning technology and the agents, evals and boring infrastructure that keep it from falling over. X/Twitter: https://x.com/dobleio canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/stop-evaluating-models-like-it-s-the-50s-alejandro-15a10fc25cd0 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/stop-evaluating-models-like-it-s-the-50s-alejandro-15a10fc25cd0.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T07:45:58.579Z' language: null children: [] videoId: O3FEoMYvUf8 durationSeconds: 1415 publishedAt: '2026-07-13T17:56:55.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 2 endSeconds: 14 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=2' heldReason: rights - ordinal: 1 text: null startSeconds: 14 endSeconds: 114 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=14' heldReason: rights - ordinal: 2 text: null startSeconds: 14 endSeconds: 114 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=14' heldReason: rights - ordinal: 3 text: null startSeconds: 124 endSeconds: 240 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=124' heldReason: rights - ordinal: 4 text: null startSeconds: 124 endSeconds: 240 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=124' heldReason: rights - ordinal: 5 text: null startSeconds: 252 endSeconds: 340 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=252' heldReason: rights - ordinal: 6 text: null startSeconds: 252 endSeconds: 340 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=252' heldReason: rights - ordinal: 7 text: null startSeconds: 352 endSeconds: 432 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=352' heldReason: rights - ordinal: 8 text: null startSeconds: 432 endSeconds: 443 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=432' heldReason: rights - ordinal: 9 text: null startSeconds: 443 endSeconds: 570 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=443' heldReason: rights - ordinal: 10 text: null startSeconds: 582 endSeconds: 663 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=582' heldReason: rights - ordinal: 11 text: null startSeconds: 675 endSeconds: 771 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=675' heldReason: rights - ordinal: 12 text: null startSeconds: 782 endSeconds: 903 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=782' heldReason: rights - ordinal: 13 text: null startSeconds: 914 endSeconds: 1036 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=914' heldReason: rights - ordinal: 14 text: null startSeconds: 914 endSeconds: 1036 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=914' heldReason: rights - ordinal: 15 text: null startSeconds: 1047 endSeconds: 1146 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=1047' heldReason: rights - ordinal: 16 text: null startSeconds: 1146 endSeconds: 1156 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=1146' heldReason: rights - ordinal: 17 text: null startSeconds: 1156 endSeconds: 1237 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=1156' heldReason: rights - ordinal: 18 text: null startSeconds: 1247 endSeconds: 1360 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=1247' heldReason: rights - ordinal: 19 text: null startSeconds: 1370 endSeconds: 1415 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=1370' heldReason: rights - ordinal: 20 text: null startSeconds: 1370 endSeconds: 1415 watch: 'https://liquidgold.cc/watch/O3FEoMYvUf8?t=1370' heldReason: rights --- # Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers Psychologists spent the last century learning how to measure something invisible and uncooperative: a human mind. AI evaluation, meanwhile, still scores like it is 1950. Count the right answers, treat every question as equal, trust the percentage (this is Classical Test Theory). We are sitting on decades of measurement theory built for exactly this problem, and we forgot to use it. Borrow it and the picture changes. Item Response Theory (or IRT, the math behind the SAT and the GRE) models every item on top of a shared scale with real error bars. That tells you which of your test items are pure noise, which are optimal, and where the knowledge gaps and unexpected behaviours are. Adaptive testing then measures the same ability with a fraction of the questions, which means private, rotating benchmarks that resist contamination instead of saturating in a month (tinyBenchmarks already hinted you can shrink a benchmark with IRT). It goes further than scoring. The statistical properties of how a model fits the test reveal something a single number never could: data leakage, the moment an agent has quietly seen the answers before. The same machinery that catches a cheating student catches a contaminated benchmark. And instead of one flat score, you get a shape: where the jagged frontier actually is, which abilities are solid and which are luck, so you know which direction to push next. You will leave this talk with a way to build evals that are cheaper, harder to game, and that tell you what your model actually learned instead of how lucky it got. This is not about handing human tests to a model. It is about borrowing a century of how to measure a mind that does not want to be measured. Speakers: - Alejandro Vidal (Mindmakers): Alex Vidal is the founder of Mindmakers, a psychologist and computer scientist who teaches humans to use AI and teaches AIs to teach humans, building adaptive learning technology and the agents, evals and boring infrastructure that keep it from falling over. X/Twitter: https://x.com/dobleio Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/stop-evaluating-models-like-it-s-the-50s-alejandro-15a10fc25cd0 ยท Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.