How this material uses LLM-as-judge In the captured conversation, LLM-as-judge means using a language model to evaluate outputs of a task rather than to perform the task. Michael Taylor's core claim: LLMs are often better at judging tasks than performing them. [^01a0f15a-e590-7f2d-8b7d-8afcf1ca33f3@890] But naive judging fails in a predictable way — grading on a holistic scale (stars out of five, essay grades) produces everything as a generous 4/5 or B+, because models “overestimate the middle” and are too nice. [^01a0f15a-e590-7f2d-8b7d-8afcf1ca33f3@955] The working recipe in the source - Reduce every criterion to binary 0/1 (Is this concise? Does it have a compelling hook?) and build roughly 20 of them per task. An aggregate of many binary judges is far more stable on re-runs than a single holistic score. [^01a0f15a-e590-7f2d-8b7d-8afcf1ca33f3@955] - Derive the criteria by using the LLM as a judge again, on comparison: feed it a good version and a bad version, ask what differs, and turn each difference into an evaluation criterion. [^01a0f15a-e590-7f2d-8b7d-8afcf1ca33f3@1042] - Score the judges themselves against known-good examples, checking false positives and false negatives — if your rewrite had a weak hook but the judge said it didn't, mark that classifier down. [^01a0f15a-e590-7f2d-8b7d-8afcf1ca33f3@955] - Optionally, optimize the classifiers with a prompt-optimization library like DSPy, which improves the generation task as well. [^01a0f15a-e590-7f2d-8b7d-8afcf1ca33f3@1042] Where it sits in the eval loop LLM-as-judge is one of three complementary eval strategies in the material, alongside backtesting a new prompt against past responses and handwriting the output you wished you'd gotten when there is no ground truth. It powers the loop that makes prompting reliable: judge drafts with binary criteria, then revise. AgentGPT's team once spent roughly $200 per commit running LLM judges in CI; the same commit costs about $2 today — evidence the cost of judging at scale has collapsed. - LLMs judge better than they perform — but only with binary criteria. [^01a0f15a-e590-7f2d-8b7d-8afcf1ca33f3@890] - Holistic scales fail: everything comes back a generous 4/5. [^01a0f15a-e590-7f2d-8b7d-8afcf1ca33f3@955] - Build ~20 binary 0/1 judges from a comparison-derived rubric, then check the judges for false positives and negatives. [^01a0f15a-e590-7f2d-8b7d-8afcf1ca33f3@1042]