---
schema: 1
type: term
title: '**Model-graded eval**'
description: >-
  A **model-graded eval** is an evaluation of an AI system's output in which
  another model, rather than a fixed rule or human reviewer, performs the
  grading. It is used for _stochastic_ software — systems whose outputs vary
  from run to run — where deterministic checks alone cannot judge output
  quality, as in the large-scale AI products this channel's material examines.
canonical: >-
  https://liquidgold.cc/channel/everyinc/explore/term/model-graded-eval-2b73abec9671
markdown: >-
  https://liquidgold.cc/channel/everyinc/explore/term/model-graded-eval-2b73abec9671.md
llms: null
parent: 'https://liquidgold.cc/channel/everyinc/llms.txt'
channel: everyinc
updated: '2026-10-23T22:12:35.346Z'
language: null
children: []
---


# \*\*Model-graded eval\*\*

Canonical: https://liquidgold.cc/channel/everyinc/explore/term/model-graded-eval-2b73abec9671

Parent: [Channel index](https://liquidgold.cc/channel/everyinc/llms.txt)

## Definition

A **model-graded eval** is an evaluation of an AI system's output in which another model, rather than a fixed rule or human reviewer, performs the grading. It is used for _stochastic_ software — systems whose outputs vary from run to run — where deterministic checks alone cannot judge output quality, as in the large-scale AI products this channel's material examines.

In this channel's material, **model-graded eval** comes up in the context of evaluating *stochastic* AI software — systems whose outputs vary from run to run, and which therefore cannot be judged with simple deterministic checks. The conversation with Notion cofounder **Simon Last** explores what it actually takes to ship and evaluate such AI software at the scale of 100 million users, spanning how knowledge-software primitives (databases, UI, permissions) combine with AI “thinking boxes,” and how interface design shaped for LLMs changes what evaluation must cover. To see the term in its fuller context — including how a team reasons about quality when outputs are probabilistic rather than fixed — explore the Simon Last conversation and its handout, which maps the moments where shipping and evaluation practices for stochastic AI are discussed.  \**Model-graded evals** address the core problem of AI products: outputs are **stochastic**, so evaluation cannot rely on fixed expected answers alone.\ \In the material, evaluation is discussed alongside **LLM-friendly interface design** and the primitives of knowledge software — all part of making AI features dependable at very large scale.\ \The term is grounded in the practical experience of **Notion cofounder Simon Last** building and evaluating AI software used by 100 million users.\