How binary criteria work in the channel’s material In the prompt engineering material, binary criteria are Michael Taylor’s fix for a common eval failure: holistic scores don’t work. Ask an LLM to grade on a scale — stars out of five, an essay grade — and everything comes back a generous 4/5 or B+. The model “overestimates the middle” and is too nice. [^42d91e03-860c-5f98-a0b1-f024b6ccea3e@890] The remedy is to break each quality you care about into a 0 or 1 question and build roughly twenty of them per task — Is this concise? Does it have a compelling hook? An aggregate of many binary judges is far more stable on re-runs than a single holistic score. [^42d91e03-860c-5f98-a0b1-f024b6ccea3e@955] Fine-grained scales can be salvaged the same way: decompose them into sub-criteria, so a “good hook” becomes separate checks like grabs attention and references something famous or credible. [^42d91e03-860c-5f98-a0b1-f024b6ccea3e@955] Where the criteria come from The binary criteria themselves are derived by using the LLM as a judge on comparison: collect original and rewritten outputs, ask what differs, summarize the differences into bullet points, and turn each bullet into a 0/1 evaluation criterion. Then score the judges themselves against known-good examples, checking false positives and false negatives. [^42d91e03-860c-5f98-a0b1-f024b6ccea3e@1042] - Binary criteria reduce each eval quality to a yes/no 0/1 judge instead of a holistic scale. - About 20 binary judges per task aggregate into a signal that is stable across re-runs. - They are generated from comparison-derived rubrics and validated against known-good examples for false positives and negatives.