What _evals_ are ## What evals are Evals are an organized system of tests and measurements used to check how well a model, tool, or product is actually performing. In this channel's material, the term comes up in two concrete settings: designing MCP servers and shipping an AI coding product like Townie. Evals in MCP server design ## Evals in MCP server design Alex Rattray argues that building an MCP server well takes real product management — talking to customers, watching how they work — plus serious engineering and an evals system that covers the whole matrix of clients (Cursor, Claude Code, and so on) and underlying models. Because the same server behaves differently across clients and models, the evals system is what lets you explore that matrix systematically rather than guessing. Evals in a shipped product ## Evals in a shipped product Steve Krouse describes the cost of skipping them: for the first three or four months of Townie, Val Town had no real evals — they would YOLO changes to production, test manually, and rely on anecdotal reports. Users would report the model got stupider over a weekend, and Krouse would take it to Twitter to ask if anyone else saw it. Once evals existed, he could check customer complaints against them — and if nothing showed up, confidently say try again. The catch he names honestly: you start only taking seriously what you can measure, and when a newer Sonnet shipped, their evals showed no difference — because none of their evals were hard enough. - Evals = an organized system for measuring how a model, tool, or product performs — not a single test, but infrastructure for checking behavior.- Rattray ties evals to MCP server quality: an evals system must span the matrix of clients and models.- Krouse shows the downside of skipping them: anecdotal panic over perceived model degradation, and no way to check complaints — and the upside: once evals exist, complaints can be checked and answered with confidence.- Evals have limits: if they aren't hard enough, they can show no difference even when a new model ships.