--- schema: 1 type: video title: 'Don''t Ship Skills Without Evals — Philipp Schmid, Google DeepMind' description: >- There are thousands of agent skills. Almost none of them are tested. They get vibe-checked with two manual runs, maybe a thumbs-up from a colleague, then shipped. You wouldn't merge code without tests — so why are we shipping skills without evals? This talk covers the full lifecycle of building reliable agent skills: what a skill actually is (and isn't), how to write one that triggers correctly, and how to build a lightweight eval harness that catches failures before your users do. ### Philipp Schmid Staff Engineer · Google DeepMind [X/Twitter](https://x.com/_philschmid) · [LinkedIn](https://www.linkedin.com/in/philipp-schmid-a6a2bb196/) · [Website](https://www.philschmid.de/) · [Blog](https://www.philschmid.de) Philipp Schmid is a Staff Engineer at Google DeepMind working on Gemini and Gemma. His work focuses on helping developers build and benefit from AI responsibly. Timeline: 0:00 Introduction: Why skills need evals 0:25 The problem with current agent workflows 1:25 Agents we use vs. agents we build 2:28 Defining a 'Skill' and progressive disclosure 3:08 Capability skills vs. preference skills 4:17 Do skills actually work? (Skillsbench data) 5:39 Model-triggered vs. user-invoked skills 6:39 Best practices for writing skill descriptions 8:30 Structuring complex, multi-layered skills 9:04 Defining goals and constraints (avoiding rigid steps) 9:56 Don't skip negative cases 10:36 Testing strategy: Evals and regressions 11:05 Removing 'no-ops' for cost efficiency 11:47 Knowing when to retire a skill 12:22 Practical example: Gemini interactions API 13:36 Building a lightweight eval harness 14:35 Using regex and LLMs as judges 17:14 Top 10 best practices summary 20:20 Homework: How to start testing your skills Viral Pull Quotes: (0:22) "You wouldn't merge code without tests—so why are we shipping skills without evals?" (1:13) "Agents are really nondeterministic. You might not know if your task fails because your skill is bad or if your task fails because it's way too challenging for the model." (9:15) "If the process or the workflow is always the same, you should not use skills. Maybe you should write a script." (11:53) "Skills are not there to live forever. Models get better, behaviors change, environments change." --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/0vphxNt4wyk-don-t-ship-skills-without-evals Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/don-t-ship-skills-without-evals-philipp-schmid-goog-bea6b8061fba markdown: >- https://liquidgold.cc/channel/aidotengineer/video/don-t-ship-skills-without-evals-philipp-schmid-goog-bea6b8061fba.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T07:45:39.202Z' language: null children: [] videoId: 0vphxNt4wyk durationSeconds: 1306 publishedAt: '2026-07-14T22:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 12 endSeconds: 113 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=12' heldReason: rights - ordinal: 1 text: null startSeconds: 12 endSeconds: 113 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=12' heldReason: rights - ordinal: 2 text: null startSeconds: 124 endSeconds: 258 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=124' heldReason: rights - ordinal: 3 text: null startSeconds: 124 endSeconds: 258 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=124' heldReason: rights - ordinal: 4 text: null startSeconds: 124 endSeconds: 258 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=124' heldReason: rights - ordinal: 5 text: null startSeconds: 258 endSeconds: 269 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=258' heldReason: rights - ordinal: 6 text: null startSeconds: 269 endSeconds: 362 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=269' heldReason: rights - ordinal: 7 text: null startSeconds: 269 endSeconds: 362 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=269' heldReason: rights - ordinal: 8 text: null startSeconds: 373 endSeconds: 454 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=373' heldReason: rights - ordinal: 9 text: null startSeconds: 454 endSeconds: 465 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=454' heldReason: rights - ordinal: 10 text: null startSeconds: 465 endSeconds: 577 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=465' heldReason: rights - ordinal: 11 text: null startSeconds: 465 endSeconds: 577 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=465' heldReason: rights - ordinal: 12 text: null startSeconds: 588 endSeconds: 692 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=588' heldReason: rights - ordinal: 13 text: null startSeconds: 588 endSeconds: 692 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=588' heldReason: rights - ordinal: 14 text: null startSeconds: 588 endSeconds: 692 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=588' heldReason: rights - ordinal: 15 text: null startSeconds: 692 endSeconds: 703 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=692' heldReason: rights - ordinal: 16 text: null startSeconds: 805 endSeconds: 817 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=805' heldReason: rights - ordinal: 17 text: null startSeconds: 817 endSeconds: 953 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=817' heldReason: rights - ordinal: 18 text: null startSeconds: 817 endSeconds: 953 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=817' heldReason: rights - ordinal: 19 text: null startSeconds: 964 endSeconds: 1044 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=964' heldReason: rights - ordinal: 20 text: null startSeconds: 1044 endSeconds: 1054 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=1044' heldReason: rights - ordinal: 21 text: null startSeconds: 1054 endSeconds: 1142 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=1054' heldReason: rights - ordinal: 22 text: null startSeconds: 1142 endSeconds: 1153 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=1142' heldReason: rights - ordinal: 23 text: null startSeconds: 1153 endSeconds: 1306 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=1153' heldReason: rights - ordinal: 24 text: null startSeconds: 1153 endSeconds: 1306 watch: 'https://liquidgold.cc/watch/0vphxNt4wyk?t=1153' heldReason: rights --- # Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind There are thousands of agent skills. Almost none of them are tested. They get vibe-checked with two manual runs, maybe a thumbs-up from a colleague, then shipped. You wouldn't merge code without tests — so why are we shipping skills without evals? This talk covers the full lifecycle of building reliable agent skills: what a skill actually is (and isn't), how to write one that triggers correctly, and how to build a lightweight eval harness that catches failures before your users do. ### Philipp Schmid Staff Engineer · Google DeepMind \[X/Twitter]\(https://x.com/\_philschmid) · \[LinkedIn]\(https://www.linkedin.com/in/philipp-schmid-a6a2bb196/) · \[Website]\(https://www.philschmid.de/) · \[Blog]\(https://www.philschmid.de) Philipp Schmid is a Staff Engineer at Google DeepMind working on Gemini and Gemma. His work focuses on helping developers build and benefit from AI responsibly. Timeline: 0:00 Introduction: Why skills need evals 0:25 The problem with current agent workflows 1:25 Agents we use vs. agents we build 2:28 Defining a 'Skill' and progressive disclosure 3:08 Capability skills vs. preference skills 4:17 Do skills actually work? (Skillsbench data) 5:39 Model-triggered vs. user-invoked skills 6:39 Best practices for writing skill descriptions 8:30 Structuring complex, multi-layered skills 9:04 Defining goals and constraints (avoiding rigid steps) 9:56 Don't skip negative cases 10:36 Testing strategy: Evals and regressions 11:05 Removing 'no-ops' for cost efficiency 11:47 Knowing when to retire a skill 12:22 Practical example: Gemini interactions API 13:36 Building a lightweight eval harness 14:35 Using regex and LLMs as judges 17:14 Top 10 best practices summary 20:20 Homework: How to start testing your skills Viral Pull Quotes: (0:22) "You wouldn't merge code without tests—so why are we shipping skills without evals?" (1:13) "Agents are really nondeterministic. You might not know if your task fails because your skill is bad or if your task fails because it's way too challenging for the model." (9:15) "If the process or the workflow is always the same, you should not use skills. Maybe you should write a script." (11:53) "Skills are not there to live forever. Models get better, behaviors change, environments change." --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/0vphxNt4wyk-don-t-ship-skills-without-evals Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/don-t-ship-skills-without-evals-philipp-schmid-goog-bea6b8061fba · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.