Where this term appears in the material In this channel's material, model-graded eval comes up in the context of evaluating stochastic AI software — systems whose outputs vary from run to run, and which therefore cannot be judged with simple deterministic checks. The conversation with Notion cofounder Simon Last explores what it actually takes to ship and evaluate such AI software at the scale of 100 million users, spanning how knowledge-software primitives (databases, UI, permissions) combine with AI “thinking boxes,” and how interface design shaped for LLMs changes what evaluation must cover. How to explore it further To see the term in its fuller context — including how a team reasons about quality when outputs are probabilistic rather than fixed — explore the Simon Last conversation and its handout, which maps the moments where shipping and evaluation practices for stochastic AI are discussed. <li>Model-graded evals address the core problem of AI products: outputs are stochastic, so evaluation cannot rely on fixed expected answers alone.</li> <li>In the material, evaluation is discussed alongside LLM-friendly interface design and the primitives of knowledge software — all part of making AI features dependable at very large scale.</li> <li>The term is grounded in the practical experience of Notion cofounder Simon Last building and evaluating AI software used by 100 million users.</li>