---
schema: 1
type: term
title: '**binary criteria**'
description: >-
  **Binary criteria** are evaluation checks reduced to a strict 0 or 1 judgment
  — a quality either holds or it doesn’t — used in place of holistic star
  ratings or grades when evaluating LLM outputs. Aggregating many binary judges
  (~20 per task) produces re-runs that are far more stable than a single
  scale-based score.
canonical: >-
  https://liquidgold.cc/channel/everyinc/explore/term/binary-criteria-54f5bea27db3
markdown: >-
  https://liquidgold.cc/channel/everyinc/explore/term/binary-criteria-54f5bea27db3.md
llms: null
parent: 'https://liquidgold.cc/channel/everyinc/llms.txt'
channel: everyinc
updated: '2026-10-23T22:12:35.346Z'
language: null
children: []
---


# \*\*binary criteria\*\*

Canonical: https://liquidgold.cc/channel/everyinc/explore/term/binary-criteria-54f5bea27db3

Parent: [Channel index](https://liquidgold.cc/channel/everyinc/llms.txt)

## Definition

**Binary criteria** are evaluation checks reduced to a strict 0 or 1 judgment — a quality either holds or it doesn’t — used in place of holistic star ratings or grades when evaluating LLM outputs. Aggregating many binary judges (~20 per task) produces re-runs that are far more stable than a single scale-based score.

In the prompt engineering material, **binary criteria** are Michael Taylor’s fix for a common eval failure: holistic scores don’t work. Ask an LLM to grade on a scale — stars out of five, an essay grade — and everything comes back a generous 4/5 or B+. The model “overestimates the middle” and is too nice. [^42d91e03-860c-5f98-a0b1-f024b6ccea3e@890] The remedy is to break each quality you care about into a **0 or 1 question** and build roughly **twenty of them per task** — _Is this concise? Does it have a compelling hook?_ An aggregate of many binary judges is far more **stable on re-runs** than a single holistic score. [^42d91e03-860c-5f98-a0b1-f024b6ccea3e@955] Fine-grained scales can be salvaged the same way: decompose them into sub-criteria, so a “good hook” becomes separate checks like _grabs attention_ and _references something famous or credible_. [^42d91e03-860c-5f98-a0b1-f024b6ccea3e@955]The binary criteria themselves are derived by using the LLM as a judge on **comparison**: collect original and rewritten outputs, ask what differs, summarize the differences into bullet points, and turn each bullet into a 0/1 evaluation criterion. Then score the judges themselves against known-good examples, checking **false positives and false negatives**. [^42d91e03-860c-5f98-a0b1-f024b6ccea3e@1042]- **Binary criteria** reduce each eval quality to a yes/no 0/1 judge instead of a holistic scale. - About **20 binary judges per task** aggregate into a signal that is stable across re-runs. - They are generated from **comparison-derived rubrics** and validated against known-good examples for false positives and negatives.