Rafal Wilinski, founding engineer at Runlayer, explains how to scale self-improving AI agents across an organization. The talk covers why agent skills matter as agents work on longer tasks, three limitations holding skills back (developer-oriented design, writing effort, client support), delivering skills over MCP from a governed server, and distilling skills from successful and failed agent runs—an approach inspired by the Voyager Minecraft paper. One reported test used a distilled skill to raise task success from 36% to 100%, illustrating an organizational knowledge flywheel that persists across model changes. # Video Identity Observations ## Source Overview - **Title:** *"From 36% to 100%: How Self-Improving Agents Write Their Own Skills — Rafal Wilinski, Runlayer"* - **Format:** A conference-style technical talk with an explicit introduction, three-part agenda, and closing pitch. ## Speaker Self-Description (Explicit Claims) - The speaker identifies himself twice as **Rafal Wilinski**, founding engineer at Runlayer: > "Hi, I'm Rafal. I'm a founding engineer at rum layer. We are making the golden path for AI at your company." - Prior experience claimed: *"I was leading Zapier agents and different AI teams"* — positioning himself as someone whose *"previous work was about making agents not just flashy demos, but reliable things in production."* - Closing re-pitch: *"We're figuring out the AI golden path for you... And this is exactly what we are building at run layer."* ## Intended Audience (Observable) - **Technical leaders and AI adoption stakeholders**, not just hands-on developers. Evidence: the speaker distinguishes between developers and *"VP of AI or AI adoption lead"* multiple times, framing governance for non-technical departments (marketing, HR, legal, finance). - **Conference attendees at an AI/tech event** — references to *"everyone here has access to them"* and *"so many people are walking today across the expo"*. ## Claimed Scope (Explicit) - Three-part structure stated upfront: improving agents via **skills**, distributing them via **MCP as a distribution layer**, and **self-improving agents/organizations**. - Enterprise-scale framing from the opening: *"a self-improving company or maybe even an enterprise that self-improves while you sleep."* - Case-study claims: an internal 36% → 100% success rate result, and *"we've been doing that internally at run layer for the past 6 months."* - Vendors-positioning claims: Runlayer *"scans thousands of skills every day"* and runs an internal *"self-improving organizational flywheel."* ## Observable Presentation Patterns - **Opinionated technical argumentation:** takes positions on industry debates (e.g., "Skills vs MCP is a false dichotomy — MCP can deliver skills"), references research (Voyager paper, METR benchmark) and ecosystem artifacts (skills-over-MCP working group). - **Product-adjacent thought leadership:** the talk builds toward Runlayer's value proposition, framing the moat as *"your own hard-won lessons... captured, compounding"* — clearly a vendor talk dressed as a technical education session. - **Humor and cultural references:** jokes about `npx skills` prompt injections, model deprecation grief ("I'm Fable. Probably just like you"), and existential AI crises — signaling fluency in current AI practitioner discourse (Twitter/Slack complaints about Opus). ## Distinction of Claims vs. Demonstrations - **Creator claims** (not independently verified here): the 36% → 100% result, 6 months of internal use, daily skill scanning, and the maturity of the skills-over-MCP ecosystem. - **Demonstrated by the material itself:** the speaker's communication style, audience targeting, argument structure, and self-presentation as a practitioner-expert founder. # Delivery Observation: Rafal Wilinski — "From 36% to 100%: How Self-Improving Agents Write Their Own Skills" ## Overall Communication Style Wilinski delivers a **conversational, technical talk** built on a three-part structure he announces explicitly, with heavy reliance on **imagined scenarios**, **rhetorical questions**, and repeated fillers ("um," "right?") throughout. The delivery reads as developer-to-developer: informal register, meme and tweet references, and persistent self-corrections mid-sentence. --- ## Recurring Linguistic Patterns ### 1. Repeated word triples / self-corrections The most distinctive observable habit is restating phrases two or three times, often with a stumble: > "all that debugging um all the all the reasoning behind that is just gone... um all that all the reasoning is just gone" > "it's w w w w w w w w w w w w w w w w w w w w wing less tokens and and less money" > "it was uh then included... the success ratio went to 100%. So yeah, that's great." > "care carefully," "prey on how to deal," "they can they can search for the right skills" ### 2. "Right?" as a recurring tag question Used to seek agreement and keep the audience engaged: > "they don't have to figure it out from the scratch, right?" > "because they are super accessible, right? At least in theory." > "is essentially solved, right?" > "hey, everyone's building the same thing, right?" ### 3. Signature framing phrases - **"figure it out from scratch"** — appears as a motif at least three times: > "they don't have to figure it out from the scratch, right?" > "if agents would try to figure out that from scratch, they are likely going to fail" > "they don't have to figure out the solution from scratch" - **"across your whole company"** — repeated when scaling from one agent to the organization: > "happening for every hard problem across your whole company" > "the source of truth across your whole company" > "deliver the knowledge across your whole company" - **"That's a dream, right?"** — rhetorical signposting of a vision before the pivot to reality ("But it almost never works that way"). ### 4. Humor via hyperbole > "That's the only command I know that installs just uh not just a skill but also free prompt injections. um a wild script that requires root permissions and someone else's API key all in one run." > "like 50% of us probably already had an existential crisis thanks to the models" > "we complain every single day on Twitter, on Slack that Opus is really dumb today, right?" ### 5. Pop-culture / internet references - Spider-Man formula adapted: *"With great power or long task horizon comes great responsibility."* - "If you haven't been living under a rock for the past 6 or 12 months." - "I'm fable probably just like you" (grieving a deprecated model — appears twice, once at [221] and again "we all miss Fable" at [853]). - Reference to a meme about `npx skills`. --- ## Teaching Habits ### Explicit structural signposting Wilinski tells the audience his roadmap and follows it: > "I want to break down my topic into three separate pieces and start by just improving agents." He then verbally bridges each section: *"Okay. So now we know how we can improve agents... But let's talk about something more interesting."* And closes the loop: *"what I promised you at the very beginning, the network effect."* ### Concrete analogy before technical definition He leads with a human scenario (the best engineer who writes down playbooks) before defining "skill" technically: > "imagine the best engineer you ever worked with... they wrote down exactly how they did that and handed over the playbook to the team" Only afterward: *"A skill to me is a playbook that essentially agent can read."* ### "Define for the audience" habit > "to make sure that we are all on the same page. A skill to me is a playbook..." > "For those who don't know, it's meter benchmark. It measures for how long agents can work independently..." ### Grounding claims in a worked example with numbers The 36% → 100% chromium-in-Lambda story is delivered as a before/after narrative: > "it's not production ready to have 36% success rate. Um so what we did is we flipped the flag for self-improving agents... and the success rate went to 100%." ### Audience-address questions as section pivots Sections frequently open with a question: *"Why is this so important? Well,..."* (used twice, at [177] and [988]). Also: *"How did they pull it off?"* and *"And how we do it?"* --- ## Tone and Register - **Hedged, first-person conviction**: "I wanted to convince you that this is actually possible"; "I believe"; "I think" appear frequently, softening claims. - **Casual interjections**: "um" pervades the transcript; also "whatnot" (twice), "Yeah" as a discourse marker ("Um, yeah, we also have the continuous learning flywheel"). - **Direct audience address**: "Hi everyone, I'm Raal"; "If it sounds interesting, let me try to introduce myself once again. Hi, I'm Rafal." — notably he **re-introduces himself** mid-talk as a re-engagement device, and again at the close ("I'm Rafal. I'm working at run layer"). - **Playful self-awareness about vendor context**: "But generally speaking, my previous work was about making agents not just flashy demos, but reliable things in production." --- ## Memorable Compressed Aphorisms A recurring teaching move is condensing a point into a quotable one-liner: > "Valuable skills are the rituals that nobody wants rediscovered." > "frontier intelligence is rented or borrowed. It can disappear." > "exploration should be turned into reusable procedural knowledge." > "The real sharpening happens when the agent fails." > "Skills can be delivering um the knowledge how to do something and tools gives us capabilities." > "it's like a developer with better attitude towards documentation." --- ## Stable Patterns vs. Isolated Moments **Stable (highly repeated, likely habitual):** - Self-correction/restatement triples - "Right?" tag questions - "um" filler density - Question-to-answer section pivots ("Why is this so important?") - Analogy-first, definition-second explanations - "across your whole company" scaling frame **Isolated or thin evidence (uncertain as habits):** - The Spider-Man line — single instance; may be prepared material rather than a verbal habit. - The meme walkthrough — one occurrence, slide-dependent. - The double self-introduction — appears exactly twice (opening and mid-talk); could be a deliberate one-off structural choice rather than a pattern. --- ## Summary Wilinski's observable delivery is that of a **practitioner-teacher**: he structures explicitly, explains via analogy and audience-alignment phrases ("on the same page," "for those who don't know"), compresses arguments into aphorisms, and maintains an informal, joke-laced developer register — with a strongly consistent verbal tic of repeating phrases in doubled or tripled self-corrections. # Video Observation Record ## Talk Overview - **Title:** *From 36% to 100%: How Self-Improving Agents Write Their Own Skills* - **Speaker:** Rafal Wilinski, identified in the transcript as "founding engineer at Runlayer"; previously led "Zapier agents and different AI teams." - **Central thesis (as stated):** Self-improving agents can be combined with network effects so that knowledge distilled from agent runs becomes a company-wide, self-improving knowledge base — a "self-improving organizational flywheel." --- ## Position 1: Skills Are the Key Accessible Lever for Improving Agents **Claim:** Among many ways to improve agents (better models, prompting, harnesses), skills are the most accessible focus area. - **Definition offered:** A skill is *"a playbook that essentially agent can read when it thinks that the knowledge is relevant."* - **Mechanism:** Described as **progressive disclosure** — the harness initially sees only a summary; if relevant, the full skill is injected into context and *"can change the trajectory of the agent."* - **Supporting reasoning:** Long-horizon benchmarks (he references the "meter benchmark," measuring how long agents can work independently "while still having a decent chance of completing a task") show frontier intelligence improving — *"it's not just vibes, it's measurable."* - **The caveat he flags:** Long task horizons cut both ways — an agent can *"dig a really really deep rabbit hole"*, spend hundreds of tool calls *"trying to defend a wrong thesis."* Hence initial trajectory matters, and skills guide agents toward the right target, saving time, tokens, and money. - **Related claim:** Valuable skills are *"the rituals that nobody wants rediscovered"* — e.g., script/feature-flag sequences for rollbacks, or wording for retaining a churning enterprise customer. He attributes many **AI POC failures** to not giving agents relevant context and playbooks. --- ## Position 2: Skills Have Three Problems 1. **Local-first and developer-centric:** The ecosystem (markdown, JSON, CLI, git) suits developers but not non-technical adopters. He adds a security jab: `npx skills` is *"the only command I know that installs just not just a skill but also free prompt injections, a wild script that requires root permissions and someone else's API key all in one run."* He claims Runlayer *"scans thousands of skills every day"* and finds *"really weird stuff inside."* 2. **No playbook often exists:** Distilling knowledge takes *"time and effort and intention and energy"* that people rarely invest. 3. **Inconsistent client support:** Strong in the Claude ecosystem; weaker elsewhere. Non-technical departments (marketing, HR, legal, finance) often use clients without skills support. --- ## Position 3: MCP Should Be the Distribution Layer for Skills - **Disputed framing he identifies:** A heated debate claimed *"thank god MCP is finally dead"* once skills arrived. His rebuttal: *"I think everyone missed the point."* - **His distinction:** Skills deliver **knowledge**; tools/MCP deliver **capabilities** — but MCP can also distribute knowledge. - **Proposal:** A remote MCP server as the **source of truth**, serving the best version of a skill per query and role, enforcing policy centrally. - **Evidence offered (company claim):** Runlayer has done this internally *"for the past 6 months and it's been working quite well."* - **Ecosystem observation:** A working group called **"Skills over MCP"** exists; *"while the exact API is not final direction is clear."* He notes the official spec leans toward MCP resources, but resource support is mixed, so *"we are going with tools."* - **Conclusion drawn:** MCP distribution solves problems 1 and 3 — unified delivery to non-technical users plus a centralized governance API. --- ## Position 4: Agents Can Improve Themselves by Distilling Runs into Skills - **Intellectual anchor:** The **Voyager paper** (2023, post-GPT-4, Minecraft). He highlights its **skill library**: discovered solutions were saved as code recipes for reuse. He distills the core idea as: *"exploration should be turned into reusable procedural knowledge."* - **Runlayer's implementation (stated):** - Successful runs with interesting traces are passed to a frontier LLM to *"simply distill a skill."* - **Failures are richer signal:** a run going sideways *"exposes the missing guardrail, the missing library, some kind of edge case."* - Successes make skills more *trustworthy*; failures make them more *robust* — all *"completely autonomously, asynchronously, without human in loop."* - **Headline result (self-reported):** Running a chromium fork in AWS Lambda with "Sonnet 4.6" (~36% success, *"not production ready"*) → after enabling self-improving agents and one successful distillation, success ratio *"went to 100%."* He also notes skills narrow the search space, reducing tokens and cost. - **Qualification:** The 100% figure is presented for a single task; he calls success rate *"the easiest number to show."* --- ## Position 5: Skills Hedge Against Rented Intelligence - **Claim:** *"Frontier intelligence is rented or borrowed. It can disappear"* — via deprecation or quality regressions (he cites everyday complaints that "Opus is really dumb today"). - **Implication he draws:** Use the best available intelligence now to *"discover and persist those useful procedures"* — building *"a catalog of how the work gets done inside our company as a collective subconscious."* --- ## Position 6: The Organizational Flywheel as a Moat - **Mechanism (described):** Group similar runs → distill a skill → pass safety checks (prompt injection, PII, *"bunch of other stuff"*) → land in skill library → served via the company's MCP gateway to every client and agent. - **Result:** A *"constantly evolving knowledge base... continuously updated with new alpha"* so no one re-solves hard problems from scratch. - **Strategic argument:** With models commoditizing and *"cost of software going to zero,"* the moat is the flywheel: *"your own hard-won lessons distilled, governed, of how your company does things."* - **Claimed benefits:** compounding knowledge that *"raises the floor for everyone,"* resistance to model deprecation, lower costs *"without ever changing a model,"* and something competitors can't copy — *"your internal culture and your identity."* --- ## Unresolved Tensions & Qualifications Preserved - The **36% → 100%** result is a single self-reported internal case, not a benchmark. - He acknowledges skills support remains inconsistent across clients and that the **"Skills over MCP" API is not final**. - He does not resolve the tension that distillation depends on the very frontier models he warns can be deprecated or "get dumber." - Security concerns he raises about the public skills ecosystem are acknowledged but not stated as fully solved — only filtered via internal checks. # Source Attribution Record ## Video Identity - **Title:** From 36% to 100%: How Self-Improving Agents Write Their Own Skills — Rafal Wilinski, Runlayer - **Video ID:** 01a15ca6-e6e1-724c-9e10-df0e0e40fb8d ## Speaker & Roles | Speaker | Role | Evidence | |---|---|---| | **Rafal Wilinski** ("Rafal" / transcribed as "Raal" once) | Solo presenter; founding engineer at **Runlayer**; previously led **Zapier agents and AI teams** | Self-introduced twice in transcript; title metadata confirms name and company | - No guests, interviewers, narrators, or second speakers appear. A closing `>> [music]` cue indicates outro audio, not speech. ## Speaker-Identified Provenance & Claims ### First-person company claims (Rafal, on Runlayer's behalf) - Runlayer is building the "golden path for AI at your company." - Runlayer has **scanned thousands of skills** (security observation). - Internally used **MCP as a skills distribution layer for ~6 months**. - Internal **self-improving agents experiment**: Chromium-fork-in-AWS-Lambda task using "Sonnet 4.6" went from **36% to 100% success rate** after enabling skill distillation. - Runlayer's pipeline: run traces → frontier LLM distills skill → checks for **prompt injection / PII** → skill library → served via MCP gateway. ### Third-party / external references cited by speaker - **METR benchmark** — cited as measuring how long agents can work independently ("it's measurable"). - **Voyager paper (2023, post-GPT-4)** — Minecraft agent with a code-based **skill library**; cited as foundational idea: "exploration should be turned into reusable procedural knowledge." - **"Skills over MCP" working group** — cited as existing; API noted as not final. - **MCP official spec** — described as leaning toward resources; speaker says support is mixed, so Runlayer uses tools. - Referenced models/tools: GPT-4, Claude ecosystem, ChatGPT, `npx skills` command, Sonnet 4.6, Opus. ### Promotional content — **FLAGGED** - Closing segment is explicitly promotional: *"We're figuring out the AI golden path for you. I'm open to talk about skills, MCP agents, whatnot."* - Core talk content also promotes Runlayer's approach/product (MCP gateway, skill distiller, flywheel) — the presenter discloses this is "exactly what we are building at runlayer." ### Industry claims (attribution: speaker, unverified) - "I believe like 50% of us probably already had an existential crisis thanks to the models." - Claims many **AI POCs fail** due to missing context/playbooks. - Claims model moats are eroding; internal distilled knowledge is a competitor-resistant moat. ## Availability & Channel Facts - No paywalls, links, or access restrictions mentioned. - No comment section or audience Q&A is present in the transcript. - Slides/visuals referenced (METR chart, meme, Voyager gameplay video) are mentioned but not visible in transcript; Voyager visual noted as unavailable: *"Unfortunately, you cannot see the video."* ## Uncertainty Notes - Transcript contains ASR errors: "rum layer," "run later," "SKS," "MCB gateway," "series of checks" → contextually Runlayer, MCP, etc. - "Fable" (repeated: "we all miss Fable") appears to be an ASR artifact of a model name (contextually likely a frontier model); not confirmable from transcript alone. - Sonnet "4.6" version string is as-transcribed and unverified. Hi everyone, I'm Raal. Today I want to talk about self-improving agents but not in a single player mode where just one agent gets smarter. I want to talk about how we can apply that concept and mix it with network effects to get a self-improving company or maybe even an enterprise that self-improves while you sleep. Um, imagine the best engineer you ever worked with or a salesperson or a lawyer. Um, the one that every time they cracked a genuinely hard problem, they wrote down exactly how they did that and handed over the playbook to the team. So whenever they face a similar problem, they don't have to figure it out from the scratch, right? And imagine that happening for every hard problem across your whole company and that happening automatically uh each time. That's a dream, right? But it almost never works that way. Um even when you crack a really hard problem um often times the knowledge um all that debugging um all the all the reasoning behind that is just gone. Oh sorry um all that all that reasoning is just gone when you move on to the next the next task or close the session. But I wanted to convince you that this is actually possible and we already have all the primitives in place to make it happen. If it sounds interesting, let me try to introduce myself once again. Hi, I'm Rafal. I'm a founding engineer at rum layer. We are making the golden path for AI at your company. Before that, I was leading Zapier agents and different AI teams. But generally speaking, my previous work was about making agents not just flashy demos, but reliable things in production. Um before we go into interesting parts, I want to break down my topic into three separate pieces and start by just improving agents. There are many ways to do that. We can use better models. We can use better prompting tools, harnesses and whatnot. But I want to focus today only just on skills because they are super accessible, right? At least in theory. um to make sure that we are all on the same page. A skill to me is a playbook that essentially agent can read um when it thinks that the knowledge is relevant from the technical perspective that the term is called progressive disclosure and it means that at the very beginning the harness or a client sees just a summary of the skill or a short description of the skill. But when it thinks that the knowledge inside the skill can be useful, it can read the whole of the skill, inject it to the main context and based on that knowledge, it can change the trajectory of the agent. Why is this so important? Well, if you haven't been living under a rock for the past 6 or 12 months, your workflow probably has changed a lot. Uh, and that's because the agents got so much better. Actually, the models got so much better. And and it's not just vibes, it's measurable. Um here you can see a chart that probably you have seen too many times already. For those who don't know, it's meter benchmark. It measures for how long agents can work independently while still having a decent chance of completing a task. And the word chance is essential here. Um we are going to go back to in a moment. But the trend is clear. The frontier capabilities, the frontier intelligence is getting better and better. They are better at multi-step deep work. There's only one caveat if that frontier intelligence is not being taken from you. And I'm fable probably just like you. But there is a catch. With great power or long task horizon comes great responsibility. If an agent can reason for much longer, it also means that it can dig a really really uh deep rabbit hole. um it can spend hundreds of tool calls trying to defend a wrong thesis reason for hours and the next agent can do exactly the same thing or maybe even do something wrong worse. Um so the initial trajectory matters a lot and that's why skills are so important because they guide the agent towards the right target otherwise you'll waste time, you will waste tokens and most importantly you'll also waste money. Um, valuable skills are the rituals that nobody wants rediscovered. That knowledge is often not glamorous. It's probably deeply embedded within your company. And if agents would try to rediscover that knowledge from scratch, like for instance, the sequence of scripts and feature flags in order to make a nice roll back or what's the exact wording and language to use when dealing with that pesky uh enterprise customer that's about to churn. Um, yeah, if agents would try to figure out that from scratch, they are likely going to fail. And often times that failure can be really costly. That's why I believe so many AIP PCs are failing because you're not giving agents the relevant context and the playbooks how to act within your company. So SKS are great, but I feel like they are not problem free. Um there are three most important problems and the first one is that they are local first and defcentric. What I mean by that is that um when you look at the skills ecosystem, you're going to find uh markdown files, JSON files, CLI commands, git repositories, and if you're a developer, that's great. But if you're um adop uh VP of AI or AI adoption lead, you need to think more empathetically about how you can roll out the AI strategy across your whole company, which includes nontechnical people, right? And the challenge here is how nontechnical people can actually create and share uh the skills. Uh I love this meme. Um this is the junior developer or nontechnical person discovering command npx skills which is the most important the most popular command to install them. That's the only command I know that installs just uh not just a skill but also free prompt injections. um a wild script that requires root permissions and someone else's API key all in one run. Trust me at run layer we are scanning thousands of skills every day and there is really weird stuff inside. Okay. Second problem is that I think we all like we all love the situation where we are facing a hard problem and there's already pray on how to deal with that situation right but often times there's there's no playbook right because it time it takes time and effort and intention and energy to reflect on a problem and leave something useful for posterity behind and the third problem is that not every LLM client support skills right if you're in cloud ecosystem, it's probably great. If you're a developer, you're probably going to manage even if the skills support is weird depending on which client you're using. Um, and once again, if you think about marketing, HR, legal, finance, those departments are probably still using chat GPT or some other weird LM client and skills might be not supported there. And sometime ago, there was this big heated debate when the skills were introduced. Um, some people proclaimed that thank god MCP is finally dead because I'm not interested in dealing with oath. I can now hardcode a bunch of CLI commands. Um, hardcode my API key inside markdown file and call it a progress, right? Um, but I think everyone missed the point. Skills can be delivering um the knowledge how to do something and tools gives us capabilities. Tools from MCPS give us capability to do those things. But MCV can be also used as a protocol to deliver the knowledge to deliver the skills. We can use MCP as the distribution layer. So instead of installing skills locally, you can create a remote MCP server which can become the source of truth across your whole company. Clients can connect to that. They can they can search for the right skills. Uh server can enforce the policy and make sure that everyone gets the best version of the skill possible depending on a query. and for instance the role that they have assigned. We've been doing that internally at run layer for the past 6 months and it's been working quite well. Um but it's no longer just an internal thing. We see that the broader MCP ecosystem is also going that direction. Um there's already a working group called skills over MCP and while the exact API is not final direction is clear. MCP can be used to deliver the skills to deliver the knowledge across your whole company. Um yeah and while the official spec is leaning towards MCP resources the support for resources is also mixed. So we are going with tools. Um and yeah if you remember the problem the first problem I mentioned and the third problem I mentioned which is um the problem that they are defentric and the lack of uh support for clients. MCP as a distribution layer fixes that because it's unified in and works in almost every single client, not just developer clients. So the problem of devel of distributing skills to nontechnical people is essentially solved, right? Plus you have now a centralized API. So as a VP of AI or like AI adoption, you have one place to govern and manage what's the knowledge that's getting shared and distributed across your company. Okay. So now we know how we can improve agents because we have um skills. We know that we can use MCP to distribute those skills across all the clients and agents. But let's talk about something more interesting which is how agents can improve themselves. And whenever I uh see a new paper or a framework speaking about this topic, I always go back to um Voyager paper. Unfortunately, you cannot see the video, but it's Minecraft gameplay in in background. So in in 2023, right after GPT4 was released, so ages ago, the question was for how long multiple agents or harnesses can explore Minecraft space, right? Um they dropped the agent into Minecraft with no instructions. They set its own goals. Agent just kept getting better without even touching the weights or or model and the prompts. And in 2023, I think it was pretty groundbreaking. How did they pull it off? Um the key thing to notice here is the skill library. So Voyager did not just explore the Minecraft space care carefully and throw the trace away. When the agent figured out something um it saved the recipe as a code that uh that consisted of the tool calls that lead it that lead the agent to the discovery. So later on when the agent decided that hey I need to create a a crafting table or a diamond pickaxe it didn't have to figure it out this whole knowledge from scratch. It could simply go to the skill uh library and retrieve it as a code as a recipe um that can be reused. And this is really important. This is the core idea that I want to carry forward is that um exploration should be turned into reusable procedural knowledge. And we started doing something very similar at run layer. Um we let the agents do the real work. And when the run succeeds and when there is something interesting in this run, there's multiple tool calls. Um we just take that trace and we ask Frontier LM to simply distill a skill, right? And while it's super simple to distill a skill from a single run, um I think we humans and agents learn the most from mistakes, right? The real sharpening happens when the agent fails. Um so a roundout goes sideways is actually the richest signal that you can get because it exposes the missing guardrail, the missing library, some kind of edge case that you haven't thought about up front. Um and every run or or pass or fail just feeds our distiller mechanism constantly making our skill better. Um the successes are making the skill more trustworthy because if the agent achieved the goal it means that hey this skill works and if it doesn't it actually makes skill more robust because now we know what's missing in here and it all happens completely autonomously asynchronously without human in loop and the initial results were great uh we had uh this task of trying to run some chromium fork inside AWS Lambda and we were trying to do that using sonet 4.6. So not the most intelligent model but sometimes it succeeded. Um but it was just you know it's not production ready to have 36% success rate. Um so what we did is we flipped the flag for self-improving agents. Um and once in one run the one of the agent invocations cracked the code our skill distiller create a reusable skill. It was uh then included in all the future runs and the success ratio went to 100%. So yeah, that's great. It's like a developer with better attitude towards documentation. Um and the success rate is the easiest number to show. But good skill is narrowing the search space. It makes the agent uh it narrows the search space and agents starts doing uh stops doing tool calls and it's exploring less branches. it's w w w w w w w w w w w w w w w w w w w w wing less tokens and and less money. But putting savings and accuracy aside, I think there's also a deeper reason why you should be distilling skills. Um the problem is that frontier intelligence is rented or borrowed. It can disappear. I think yeah, we all miss Fable. It was a great model, right? It can deprecate it or it can get dumber. Like we complain every single day on Twitter, on Slack that Opus is really dumb today, right? So while we have the access to the best version of that intelligence, we should be using that to discover and persist those use useful procedures. We should be building a catalog of how the works gets done inside our company as a collective subconscious. And how we do it? Well, I believe we already have all the pieces of the puzzle here. Um, we have skills as a way to retain procedures and improve agents and guide models. We have MCP as a unified distribution layer for those skills. And we also have a central repository, the knowledge base for all of those skills, which is constantly being updated and fed with new data. Um, yeah, we also have the continuous learning flywheel for one agent. But what's interesting is what I promised you at the very beginning, the network effect. Um, so what happens when you multiply that by every agent and every single client in the entire company? At run later, we call it selfimproving organizational flywheel. Um, we group a bunch of similar runs and try to distill a skill. It passes through our various series of checks like whether it's not containing some kind of prompt injection or maybe PII um and bunch of other stuff. Once it's good, it's landing in our skill library and thanks to our MCB gateway, it's being served to each and every client and agent within the company. What we end up having is a constantly evolving knowledge base which is always up to date, continuously updated with new alpha fed with new age edge cases. So once one agent figure outs a really hard problem, it's getting distilled into a skill persistent in all or organizational library. So whenever someone else um has the same problem in the future, they don't have to figure out the solution from scratch. They don't have to know uh what's the exact sequence of this magical commands that are going to restart the database or roll back the production. They can simply um reach out to the playbook. they have. And why is this so important? Because I believe like 50% of us probably already had an existential crisis thanks to the models. And the thinking goes along the lines of u okay so models are becoming commodity. Everyone here has access to them. Um cost of software is going to zero. Everyone is can be rebuilding my product. Um at least on the surface. In fact, so many people are walking today across the expo and they are thinking, hey, everyone's building the same thing, right? So, what is my moat? Um, and I believe the mode can be in building this flywheel and distilling the knowledge through this flywheel because it's like your own hard one lessons distilled governed of how your company uh does things you know and it's captured compounding raising the floor for everyone across the company giving them access to the best knowledge available. Um, it is also a way to be resistant against model intelligence deprecation and it's a great way to lowers your costs without ever changing a model. So yeah, that's probably one thing that competitor can't copy your internal culture and your identity and yeah that's pretty much it. Uh, thank you for coming if it resonated with you. I'm Rafal. I'm working at run layer. We're figuring out the AI golden path for you. I'm open to talk about skills, MCP agents, whatnot. And this is exactly what we are building at run layer. Thank you. \>\> \[music\]