Talk by Philipp Schmid (Staff Engineer, Google DeepMind) on building and evaluating agent skills. Covers defining what a skill is, capability vs. preference skills, model-triggered vs. user-invoked skills, writing effective skill descriptions, structuring complex multi-layered skills, defining goals and constraints, negative test cases, eval and regression testing strategies, removing no-ops for cost efficiency, skill retirement, a practical example using the Gemini interactions API, and building a lightweight eval harness with regex and LLM-as-judge approaches, concluding with best practices and guidance on getting started. # Channel Identity Observations — "Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind" ## Explicit Self-Description by the Speaker - **Speaker affiliation (stated):** > "My name is Philip. I'm based out of Germany. I'm part of the Google DeepMind team mostly working on Gemini API and agents." - **Stated role scope:** Works on the **Gemini API and agents**, indicating the talk draws on internal Google DeepMind practice: > "we have tests or evals alongside every skill we have internally at Google DeepMind." ## Claimed Scope of the Content - The talk claims to teach **evaluation practices for agent skills** — specifically: > "we are going to talk about why you should not ship skills without eval" - It claims to cover: - What skills are and how they work (progressive disclosure, model-triggered vs. user-invoked) - Eight tips for writing good skills - A practical example: building an eval harness for a **Gemini Interactions API** skill with **117 test cases** - Ten best practices and homework for the audience ## Intended Audience (Observable) - **Developers who already use coding agents** — demonstrated by the opening poll: > "raise your hands if you use coding agents to write code... And do you have evals for those skills?" - **Engineers building agents for customers**, distinguished from agent users: > "When you build an agent inside your application for consumer or customers, they have no idea about what a skill is." - Audience assumed to have **hands-on familiarity with skills**: > "every one of us knows hopefully by now what a skill is. It's like basically really a folder with a skills.md file in it" - Practical, action-oriented framing suggests a **conference/session audience** with laptops open (speaker references "if you have your laptop open"). ## Demonstrated Presentation Patterns - **Interactive live-session style:** hand-raising polls, direct audience address, closing homework assignment ("pick the most used skill and write five test prompts"). - **Data- and benchmark-driven argumentation:** repeatedly cites **Skill Bench** (50,000 skills analyzed; ~15% average performance improvement; 100 tasks; 1.1 update with leaderboards open to community contributions). - **Crediting external sources:** attributes the "no-ops" tip to a third-party educator ("I have to give all credits to Matt... he's a great AI educator and you should definitely follow him"), and points to a GitHub skill repository. - **First-party engineering evidence:** describes internal Google DeepMind eval infrastructure (JSON test cases, regex asserts, LLM-as-judge, regression tests blocking merges) and a concrete result ("improved the performance up to like almost 90% for generating valid interactions API code"). - **Claims vs. demonstrated distinction:** the ~90% improvement and internal practices are **creator claims**; the demonstrated pattern is a structured, tutorial-style talk with checklists and examples supporting the claims. ## Title–Content Alignment - The title's promise ("Don't Ship Skills Without Evals") is directly fulfilled: the talk closes with > "So, don't ship skills without evals. Thank you." — indicating a didactic, thesis-driven presentation format consistent with a technical conference talk. # Delivery Observations — Philipp Schmid, "Don't Ship Skills Without Evals" ## Tone & Register - **Conversational, workshop-style delivery**: frequent filler words and hedges — "I mean," "like," "you know," "um" — e.g., "It's like basically really a folder with a skills.md file in it." - **Friendly, slightly humorous opener** aimed at audience self-recognition: > "Everyone uses skills, no one has evals. Hopefully we can fix that today." ## Audience Interaction - Opens with a **live show-of-hands poll** (repeated three times in sequence): > "raise your hands if you use coding agents to write code... do you use skills with it? ... Do you have evals for those skills?" - Uses the poll result as the talk's hook, then returns to audience-directed tasks at the close ("I have some homework for you"). ## Recurring Phrases & Verbal Habits - **"Very important"** as an emphasis marker, used repeatedly: > "And like very important is like why checks fail in productions." > "So, very important is the the why and the how for the model." - **"Last but not least"** appears at least twice: > "And then last but not least, uh 10 examples for best practices for skills." > "And then last but not least, uh 10 examples for best practices..." - **"So" as a pivot word** starting a large share of sentences and transitions. - Repeated **"you should definitely..."** constructions for recommendations: > "you should definitely look at the skill after our session." > "you should definitely follow him." - Occasional **self-corrections and restarts**: "I going back to a skills bench which has an update of 1.1"; "the we don't use YAML, but it's like just as like uh an example." ## Teaching Habits & Structure - **Numbered, list-driven organization**: explicitly frames sections as counts — "eight... examples on how or tips on how you can write good skills," "tip seven," "10 examples for best practices for skills." - **Contrast framing** as a core explanatory device: > "there's a big difference between the agents we use and how we use skills and the agents we build and how our customers might want to use skills" - Also: "capability skills and preference skills"; "directives instead of essays"; "outcomes, not paths." - **Concrete examples with named tools/products**: Cursor, Claude Code, anti-gravity, Gemini CLI, TypeScript and Python, AWS/Google Cloud/Azure. - **First-person case study** near the end (the Gemini Interactions API skill, 117 test cases) grounding the advice in his team's practice, including quantified results ("improved the the performance up to like almost 90%"). - **Attribution of others' work**: credits an external contributor — "tip seven which is quite new and I have to give all credits to Matt... he's a great AI educator." - **Actionable homework close**: the talk ends with a concrete to-do list for Monday, mirroring the poll-style opening ("pick uh the most used skill and write five test prompts"). ## Quantification Style - Habitually cites **specific numbers as evidence**: "roughly 15%," "below 500 lines of words," "50% of the failures," "10 to 20 skill eval samples," "run the to six uh trials per case," "117 test cases." ## Vocabulary Patterns - Consistent technical register: **skills, evals, harnesses, traces, no-ops, regression tests, LLM as a judge, ablation test, model-invoked / user-invoked**. - Repeated cost/performance framing: "save the cost," "money you will like spend," "very cheap to run." ## Uncertainty Notes - No curated metadata was provided; observations rest on the transcript alone. - Filler-word frequency and self-corrections suggest spontaneous delivery, but whether this reflects the live talk or transcription artifacts cannot be determined from text alone. - Audience responses (laughter, applause) are only minimally observable — only "[applause]" at the end is recorded. # Observation Document: "Don't Ship Skills Without Evals" — Philipp Schmid, Google DeepMind ## Speaker - **Philipp Schmid** (self-identified, based in Germany, Google DeepMind, working on Gemini API and agents). ## Central Thesis > "Don't ship skills without evals." - Audience poll (his observation): nearly everyone uses skills with coding agents, but very few have evals for them. ## Why Evals Matter - **SkillBench** indexed ~50,000 skills; almost none had evals — mostly AI-written and untested. - **Non-determinism problem**: when a task fails, you can't tell whether the skill is bad or the task exceeds model capability — evals distinguish these. - SkillBench 1.1 findings (open and closed models, multiple harnesses): - Skills improve performance by **~15% on average** across ~100 coding/productivity tasks. - **Human-written skills outperform AI-generated ones**; AI-generated skills can negatively impact performance. - Skill files should be **under 500 lines**. ## Agents We Use vs. Agents We Build - **Agents we use** (Antigravity, Cursor, Claude Code): the user is the engineer with skill context; they notice a missed skill trigger and can reprompt or use slash commands. - **Agents we build** (for consumers): customers have no idea skills exist — they won't write "use the refund skill." Only **model-invoked skills** apply here, which is where eval effort should focus. ## What a Skill Is - A folder with a `skills.md` file plus assets; uses **progressive disclosure**: 1. Title + description (always in context; determines triggering). 2. Skill body with instructions. 3. Reference files for deep context. - **Two kinds of skills**: - **Capability skills** — teach what models can't yet do consistently (log tracing, React scaffolding). Temporary; evals tell you when to retire them. - **Preference skills** — durable, encode team/company workflows and style; protected by evals because foundation models won't integrate domain-specific knowledge. ## Triggering - **Model-triggered** vs. **user-invoked** skills. - Schmid claims people **underestimate user-invoked skills** for workflow tasks (PRs, staging docs) — anything runnable as a script should likely be user-invoked. - Customer-facing agents rely only on model-invoked skills. ## Eight Tips for Writing Skills (Schmid's list) 1. **Description is critical** — include *why*, *how*, *when*; a weak description over- or under-triggers. 2. **Write directives, not essays** — e.g., "Use the Interactions API if working on a chat application." 3. **Keep it lean and layered** — description costs tokens on *every* invocation; platform-specific instructions (AWS/GCP/Azure deploys) belong in reference files. 4. **Set the right level of freedom** — fixed step-by-step workflows should be scripts, not skills; define goals and constraints instead. 5. **Don't skip negative cases** — e.g., "web development" over-triggers; specify "React components or Tailwind CSS only." 6. **Test early** — 10–20 prompts: ~5 happy path, ~5 non-triggering, plus production traces ("nothing is better than real-world data"). 7. **Remove no-ops** — attributed to **Matt** ("great AI educator"); AI-generated skills contain instructions that don't change agent behavior (e.g., "make code easy to read"). Matt published a skill for killing no-ops. 8. **Know when to retire** — run evals with/without the skill; retire when the model performs equally without it, saving token cost and maintenance. ## Practical Example: Gemini Interactions API Skill - The **Interactions API** was released after Gemini's last training, so the model lacks knowledge of it. - Team built a skill; created **117 test cases** from real user data, synthetic cases, and user feedback (e.g., model using Gemini 2.0 instead of 3.0). - Result: improved valid Interactions API code generation **to almost 90%**. - Implementation: a **JSON file** (prompt, language [TypeScript/Python], `should_trigger` flag, expected checks) plus a **Python script** running Gemini CLI. - **Regex asserts** suffice for most checks (correct SDK, model, methods, old patterns) — cheap to run, no LLM judge needed. LLM-as-judge with a rubric is available for complex skills needing full-trace review. ## How Google DeepMind Evaluates Skills Internally - Evals live alongside **every internal skill**. - Each test has multiple cases with prompts, run in **clear/isolated workspaces** (configurable files, startup commands installing libraries). - Uses script/data evals (regex over traces) and LLM-as-judge expectations (did it trigger, did it run a command). - **Regression gating**: every skill diff runs evals; changes don't merge unless tests improve or new evals are added. ## Ten Best Practices (summary of his claims) - **Description quality matters most**: ~50% of failures he's seen came from incorrect skill triggering due to shallow user prompts. - Directives over passive information; include negative tests; start small (10–20 samples beat nothing). - **Test outcomes, not paths** — whether the task is achieved matters, not whether the skill loads on turn one. - **Isolated runs** — coding agents can "cheat" by pulling context from previous chats/executions. - **Multiple trials** (he suggests running several per case) because models are non-deterministic; measure reliability. - **Test across harnesses** — a skill good with Gemini may fail with Codex; include Cursor users, etc. - **Keep evals after retiring a skill** to watch for performance degradation and reintroduce if needed; model updates let you retire skills surprisingly fast. ## Homework - Pick the most-used skill; write five test prompts (optionally ask the coding agent to analyze trajectories). - Build the eval harness (JSON/YAML + script). - Remove no-ops (cost savings even if eval performance is unchanged). - Reference Matt's "writing great skills" on GitHub. - **Run ablation tests** (skill on vs. off) to know when to retire a skill. # Speaker Attribution & Provenance Record ## Video - **Title:** Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind - **Video ID:** `01a15ca8-a4fa-7b63-bee6-bea6b8061fba` ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Philipp Schmid** (transcribed as "Philip") | **Presenter / sole speaker** | Self-introduction: *"My name is Philip. I'm based out of Germany. I'm part of the Google DeepMind team mostly working on Gemini API and agents."* | | **Audience** | Unnamed participants | Interactive hand-raising at the opening; applause at close | | **Matt** (last name not given) | **Cited third party — AI educator** | Credited for tip on removing no-ops from skills; referenced tweet, a published skill, and a GitHub skills repository | No narrator, guest, or additional identified roles appear. Curated metadata was not supplied beyond the title. ## Third-Party Claims & Cited Sources - **Skill Bench** (benchmark, update 1.1): claims skills improve performance by **~15% on average**; covers ~100 tasks; analysis of ~50,000 skills, most AI-written without evals; findings that **human-written skills outperform AI-generated ones** and skill files should stay **under 500 lines**. - **Matt's tweet / skill / GitHub repository**: AI-generated skills contain many **no-ops** (instructions with no behavioral effect). - **Gemini Interactions API case study** (speaker's own work at Google DeepMind): 117 test cases from real user data, synthetic cases, and user feedback; result of ~90% improvement in generating valid API code. ## Promotional Segments - Speaker recommends **following Matt** as "a great AI educator" and his GitHub resources — flagged as endorsement, not sponsor segment. - References to Google DeepMind internal eval practices and the Gemini CLI/product are descriptive of the speaker's employer, not flagged as explicit promotion in the transcript. - No other promotional or sponsored segments identified. ## Unavailable Evidence Channels - **Comments:** Not available / not supplied. - **No curated metadata** (description, timestamps, links) was provided; the referenced blog post and GitHub links were mentioned but not included. - Matt's full identity and link URLs are not stated in the transcript. \>\> Yes, so hi everyone. My name is Philip. I'm based out of Germany. I'm part of the Google DeepMind team mostly working on Gemini API and agents. And we are going to talk about why you should not ship skills without eval. And maybe before we start, I need a little bit of your help. So if you could raise your hands if you use coding agents to write code. So yeah, hopefully every every hand goes up, right? And do you use skills with it? Okay. Do you have evals for those skills? Okay, yeah, that's um not a lot of hands. Everyone uses skills, no one has evals. Hopefully we can fix that today. And like very important is like why checks fail in productions. And um Skill Bench is a very popular and nice like eval or benchmark which um index like over like 50,000 skills from like it up and like try to look into him and almost none of those skills had evals. Most of them were AI written um not really tested and it's very hard to know if your skill is good or bad because like agents are really non-deterministic. So you might not know if your uh task fails because your skill is bad or if your task fails because it's way too challenging for the model. So um very important um before we go into it there's I want to like really make sure that we know the difference between the agents we use and the agents we build. Um most of us use agents for writing code, doing productivity work. That's the agents we use. It's like anti-gravity, cursor, Claude code. And there are you are the engineer and you have context about skills, right? If you write some prompt to I don't know, like help me build a new Gemini API feature. And if your agent does not invoke the skill on the first time, you will notice it very quickly. You stop your your task and reprompt it or like use slash commands for for triggering those skills. When you build an agent inside your application for consumer or customers, they have no idea about what a skill is. They don't start their prompt with use customer support skill to like help me refund or use refund skill to help me solve my problem. So, there's a big difference between the agents we use and how we use skills and the agents we build and how our customers might want to use skills in like the context there. And what is a skill? I mean, every one of us knows hopefully in by now what a skill is. It's like basically really a folder with a skills.md file in it and then some additional assets to make that skill really work. And the big difference with skills is that they work on progressive disclosure. So, most of the skills start very small. So, you have the title and a description. The description is normally part of the model's context. So, the model knows when to use the skill. Second layer is we have a skills body with more instructions, more details, and hopefully more references to external files. And then you can really go deep in those reference files where there's all of the context the model needs to discover to to solve the task. And I like to differentiate between two kinds of skills. So, they are capability skills and preference skills. Capability skills teach models something they cannot do consistently at the moment. Maybe it's like, I don't know, like tracing some logs, creating a new React app. And those capability skills are temporary. So, the better our model gets, the more likely it is that we can remove those skills. And Evals will tell us when we can retire skill and when not. And then we have preference skills. Those are more durable, mostly encode some references. So, if you have a specific workflow in your team or a specific style language or other preferences which are very specific to your company, Um will have or create preference skills and those uh preference skills are then protected with e-walls where because most of like the foundation models might not uh integrate the knowledge which is very specific to your use case or your domain. And preference skills are very valuable, so we really want to make sure that those are working and we don't like update our agents to uh degrade performance. So, do skills work? Yes, they do work and I going back to a skills bench which has an update of 1.1 which has evaluated all kinds of open and closed models in different harnesses showing that skills on average improve the performance by roughly 15%. Skills bench covers around 100 different tasks uh based on like coding and also productivity across different languages. It's uh openly available and they have a very nice website, a very nice leaderboard, are also very open for uh community contributions. And then they did a second analysis on self-generated or AI-generated skills, right? It's very easy if you are in a coding agent and you work on something, tell the model create a skill and then it writes a skill.md file. We maybe look at it very closely. It roughly covers what we want to do and then we just accept it and start using it. And what I found out is that human-written skills are the best we can provide. Uh AI-generated skills can uh impact performance negatively. And that skills or skills.md files should be below 500 lines of words. So, if you have your laptop open and have a skill available, if you open that and if it's above 500 lines, you should definitely look at the skill after our session. And um the last topic about what is a skill and how a skill works, uh we have different ways of triggering our skill, right? We can have a model-triggered skill meaning uh based on the context and the description, the model decides to use or read a skill to, uh uh, get more context to solve a task. And then there are user-invoked skills. And I think people underestimate how powerful user-invoked skills are. Um, and they most of the time just accept, uh, the overhead by pro- like adding it into the context. I have like many user-invoked skills for more workflow type of tasks like creating a pull request, uh, staging documentation, and all like of the very uh, normal dev work which could be run in a script should most likely be a user-invoked, uh, skill. And when you build agents for customer, you don't have those user-invoked skills. We are only working in the model-invoked skills, and that's where we are like focusing on for the small eval section we are going to look in a second. So, writing skills, um, is an important topic. Uh, we're going to look at eight, um, examples on how or tips on how you can write good skills. And most importantly, if you work with model-invoked skills is the description because the description are most of the time two sentences we provide to the system instruction to help the model know when it should use a skill or not. And it's bad if your description is too weak because then it might trigger too often or the it might not be triggered if you need it. So, very important is the the why and the how for the model. So, why it should use that skill and then how it should use that skill. Um, very common is like use that skill if you are working on a React application, for example. And then, of course, the when. And we should write directives instead of essays. So, we should not say something like, "Hey, the Interactions API is recommended for multi-chat, um, multi-chat because it handles like session state and it's like where you should be way more directive like use the Interactions API if you're working on like a chat application. So, you need to give the model like clear instructions and directives on when it should use the skill and how it should use the skill. And similar to what we have seen in the skills bench results, we should keep the skill lean and layer information. So, the description is the cost you always pay on every model invocation. So, on every model call, the description is part of the model context. So, you always pay that 100 200 tokens cost and you don't want to have a super long description because then you always have to pay that. When you have a very long skill MD file, it will be always read into context when the model decides to read the the skill or to use the skill. Uh which can be expensive as well. That's why we want to keep it as concise as possible, but still include all of the reference and details for the model to solve the task. And then of course, the layer three is like we can have those reference files where the model needs to like um go really deep into a very specific task. And a good example for this is like if you are working in like maybe a multi-cloud environment and you have a skill to deploy your application, you might need instruction to deploying to AWS and deploying to Google Cloud. Those should not be part of your skill MD file. Those should be references. That you have a reference for AWS, reference for Google Cloud, maybe a reference for Azure so that the model can basically explore based on the context where it should go to get all of that information. Then we should set the right level of freedom. Um I see many people very clearly describing the exact workflow in a skill. Step one, go there. Step two, do this. Step three, do this. If you have those type of use cases, you should not use skills. You maybe you should write a script because if the the process or the workflow is always the same, you don't need to waste models and tokens for that exercise. You can create a script. You can tell the model use that script to run a specific workflow. So, rather define goals and constraints. So, if you need to like deploy to your update or stage your documentation, describe how the model can do that. Or like for your database updating a config, you should not say like read the config, update the port, and then like deploy again. The model knows what to do. Just like hey, if we need to change the config, here's the file, make the change. Then uh don't skip negative cases. So, we always look at the when we want to use this skill, but most of the time we don't look at when we don't want to use this skill. So, if we have a description for our skill which says use it for web development tasks, it might over trigger. Maybe you work with React, maybe you also work with Angular, and the model always loads the skill if you are working in like a web development environment, but if you are very specific for like hey, only use that skill for React components or for Tailwind CSS, then the model knows hey, that's very specific for one to use. And with Evals, we can also identify those. Um And then test early. So, that's what we are going to look at. We should really try to test when you create a new skill. Always try to create 10 of 20 prompts. I like to create five for like the happy path. So, when do I want to use that skill? Five when I don't want to use that skill just to make sure the model is not over triggering the skill and confusing itself. And then if you have already some customer or production traces, try to include those as well because nothing is better than than real-world data. And then tip seven which is quite new and I have to give all credits to Matt. So, if you don't know Matt, he's a great AI educator and you should definitely follow him. He published a tweet and also a skill on like killing all of the no-ops. And what he found is that AI generated skills tend to include a lot of no-ops. And no-ops basically is an instruction which does nothing to change the agent's behavior. It's like before making an implementation easy to read. Like the model knows how it when it should make something easy to read or write clear high-quality code. I mean, like that's what we expect from the model to do without telling it really. So, um definitely look at those no-ops. He have has published a very good skill in in his like skills repository. Uh and then last but not least, uh know when you should retire skill. Um skills are not there to live forever. Models get better, behaviors change, expectation change, um the environment changes. So, um always try to run evals with and without the skill enabled. And if the model achieves the performance without even like triggering the skill, you know you can retire that skill, save the cost uh for your tokens, and then also um don't keep like it redundant. So, save cost at the end and maintenance also as well. And to look at a little bit of a practical example and also how you can create your own small eval or eval harness for skills. Um earlier this year we wanted to create a new skill for the Gemini Interactions API. So, the Gemini Interactions API is our new interface for working with Gemini models and with agents. And the Interactions API was released after the last training of Gemini. So, the model or Gemini 3 and like 3.1 or even 3.5 has no context about what is the the Gemini Interactions API. So, we decided, "Okay, let's look uh at creating a skill to help the model create good code for the Interactions API, to use the latest models." And to do that, we created 117 test cases. Those are uh based on like data we see uh from real users trying to generate uh Gemini code from um synthetic generated uh test cases, and also from like feedback we see people like, "Hey, the model is like using Gemini 2.0 even if we are already on 3.0." And the the end result was that we improved the the performance up to like almost 90% for generating valid interactions API code with the latest Gemini models. And to do this, we basically only needed like two very simple uh assets. So, one of that was a JSON file with all of our test cases. And it's very like no clear structure. It's like, "Hey, we have a prompt." That's basically what we expect the user to provide. We have a language because we wanted to test the skill against TypeScript and Python. Uh we have a should trigger. That's basically there to tell us if the agent should read the skill or not read the skill. And then we have different expected checks. Uh we look at them in a little bit. Those are basically uh very simple asserts uh for that prompt if it should trigger or not. And then we have a very basic Python script which runs a coding agent. In this case, it was the Gemini CLI which uh passes the output and returns runs it so we can like take a look at the the outcome whether we have valid code for the interactions API or not. And uh most of the tests or evals for skills can be regex. It's like very amazing how good of regex you can write using coding agents. And it's really for us it was all about, "Okay, do we use the correct SDK? Do we use the correct um model? Do we use the correct methods? Do we use any old patterns?" And we created um very basic asserts for all of those cases, which are very cheap to run. So, we can run our skill against the evals many times. So, if a new model releases, we have a very easy way to update those asserts to the latest model IDs. And it's very cheap to run for it well because we don't need to use like LLM as a judge. But of course, you can use LLM as a judge if you have like more complex skills which need to look at the whole traces or the whole steps taken. And a very easy case is like you just create LLM as a judge with a rubric on like what you want to look at, and then like take the output, put it through the LLM as a judge, try to get a pass or a fail, and then if it fails, look at the data, and then like try to um improve your skill based on that. And that's also how we now uh eval skills at Google DeepMind. So, the we don't use YAML, but it's like just as like uh an example. Uh we have um tests or evals alongside every skill we have internally at Google DeepMind. Um every test has multiple cases with like a prompt. Uh we all run them in like clear workspaces, so you can define your workspace or environment if it should include additional files like your application environments. You have uh startup commands, which basically preloads or installs libraries into the environment. And then you have script evals or data. Those are those regex where we look at all of the traces to see what the skill triggered, was a certain command run, was a certain CLI run. And then we also have LLM as a judge where we have some expectations, which are basically matched against like hey, did it trigger the skill, did it run a certain bash command to like also evaluate it. And we run them on every change to the skill. So, if a change happens to or like a diff to the skill file, the eval will be run, and there will also be a result, and the change will not be merged if it is not improving the test cases. So, we always have those regression tests for every change to the skill, and you can only change the skill if it improves the eval or add new evals. And um yeah, that that's how we we we basically manage it. And then last but not least, uh 10 examples for best practices for skills. You don't need to take photos. They are in the blog post I can share later. So, um the I mean, we had it many, many times. The the skill skill description is very important. Uh we have seen 50% of the failures uh because the skill was not triggered correctly because the prompt of the user was not uh detailed enough for the model to understand, "Hey, I need to use that skill to solve that task." And especially if you build agents for others, they are not aware of the skill descriptions you have for your model and for your skill. So, they might write something very um shallow and then the model needs to know, "Okay, I need to trigger that skill." Um we should write directors over passive information. So, we should always think about it. You should tell the agent what to do or not what to do and not just like, "Hey, if you feel happy today, please use the skill." Um include negative tests. Uh we always forget negative tests. Start small. Even like 10 to 20 skill eval samples are better than nothing. You will be surprised on how much you will find even from like five to 10 examples. And then like definitely create outcomes, not paths. Uh we don't want to test if the model loads the skill on like the first turn. We really want to test if it can achieve the task based on the prompt. And if it loads the skill, it loads the skill. If not, then not. If it um loads the skill after five turns, that's also okay. Then we want to have isolated runs because coding agents are very good at finding or cheating. So, if you run inside uh your existing environment, it might look up previous chats or it might look up some other executions and then like try to cheat it and get the context from the skill without even using the skill. Then definitely run more than one trial when running evals. Like agents, our models are non-deterministic. Maybe the first one works, the second one doesn't. So, always run the to six uh trials per case and to measure reliability Uh um test across different harnesses if you work with like or if you have employees or um people working with like different harnesses, not only just evaluate against Claude or anti-gravity. If you have people working with cursor, try to include them as well because agent harnesses behave differently and of course model behaves differently. So, maybe your skill is very good with a Gemini but very bad with Codex and then you have uh customers, consumers using your harness with Codex and then it fails. And then um create your evals. So, if your um model is good enough that it doesn't need the skill anymore, keep that eval. You don't need to throw that eval away because you throw the skill away. You can keep that eval to make sure that the model or the agent keeps the performance and as soon as you start seeing some degradation, you can reintroduce the skill. You can maybe tweak some other tools or pieces to keep like the the performance up and then really detect when you can retire skill and you will be very surprised with all of the model updates how fast you can retire skill which you might need it like six months ago but not today anymore. And I have some homework for you. So, if you are back from holiday on Monday, um pick uh the most used skill and write five test prompts. Uh you can also use your coding agent and ask it to see look at your trajectories which are my most used skills and then try to create some some skills. It's you have seen it's like very easy to write your eval harness. It's like a JSON or YAML file and then like some Python script which runs your coding agent or your agent harness and then like look at the outcome. Definitely uh try to look at the removing no-ops. Maybe it does not change the eval performance but it helps you save cost because all of the tokens which are not helpful or not changing the agent behavior are money you will like spend. So, look at um writing great skills uh from Matt. It's you can find it on on GitHub and then also run ablation test. So, run always evals with your skill loaded and without your skill loaded. Only that way you will know when you can retire skill or if a skill is really helpful for your performance. So, don't ship skills without evals. Thank you. \>\> \[applause\]