Harald Kirschner, Principal Product Manager for VS Code at Microsoft, explains how the team moved from monthly to weekly releases after 10 years — attributing the shift not to AI alone but to rebuilding the development system around AI agents. He covers three stages: shipping faster (agent-ready codebases, skills, TypeScript Go builds 10x faster, letting agents drive the app), holding quality (mandatory AI code review, AI issue triage, error stacks turned into auto-fix PRs, staged rollouts), and learning faster (VSC-Bench evals, daily prototyping, smaller squads). Talk points include agent-written code commit survival rising from 55% to 86%, CI bottlenecks with 20 concurrent agents, self-correcting loops via Playwright, 51 billion daily telemetry events driving auto-filed issues, and a 5-character 'hello world' eval costing one model 70x more tokens. Recorded at the AI Engineer World's Fair 2026 in San Francisco. # Channel Presentation Observations: VS Code Agent-Driven Release Talk ## Source Basis - **Title and transcript** were provided. - **No curated channel metadata** was supplied, so no channel-level identity context is available. - All observations below concern this single video, a conference talk, and do not describe a channel as a whole. --- ## 1. Explicit Self-Description These are statements the speaker makes about themselves or their work. They are **creator claims**, not independently verified facts. - **Role at VS Code:** The speaker says, "I'm working on VS Code," and later describes the role in the first person: "can I as a PM effectively VIP code in the VS code repo which I do um." - **Adjacent work:** "on my site I work on iOS apps," with a mention of using Xcode MCP. - **Speaking context:** The opening refers to this being "the rerun of a talk that she gave yesterday upstairs," and the closing says "Come by code booth. Happy to chat now." --- ## 2. Intended Audience (Inferred from Content and Setting) The audience is addressed directly and is framed around engineering practice rather than product marketing. - **Software engineers and engineering leaders** evaluating how to change their own workflows. Evidence includes: "how you can encode your expert knowledge," and "try these things, find out where your bottlenecks are." - **Teams with large codebases and CI/CD pipelines**, as shown by the discussion of build times and agents "hitting the same bottleneck." - **Conference attendees**, as shown by the booth invitation and the live Q&A-style audience check: "Who has filed a GitHub issue? Who has had problems in VS Code before?" The speaker explicitly separates this audience from product-feature shoppers: "it's not about how we build agents into VS Code, which we can try out anytime." --- ## 3. Claimed Scope The speaker states the scope of the talk directly: > "this is about how we use agents to build the product itself." The claimed topics, in the order presented, are: - **Release cadence:** Monthly to weekly releases. - **Metrics:** "code survival" tracking, with a reported rise from **55%** (GPT-4.1, as transcribed) to **86%** (Claude Opus 4.6, as transcribed). - **Codebase readiness:** `agents.md` files and reusable skills. - **Build speed:** A reported **10x** build improvement from moving to TypeScript Go. - **Feedback loops:** A component browser for UI screenshots, a Playwright-based `/launch` skill, and a self-correcting loop. - **Code review and triage:** AI-assisted review, issue filtering, and error-stack-to-PR automation. - **Staged rollouts** and **evals** (VSC bench). - **Prototyping** as a way to speed up design conversations. The speaker also limits the scope: "You're not token maxing. You're not spending every day using AI." --- ## 4. Demonstrated Presentation Patterns These are patterns visible within this one transcript. ### Concrete Metrics and Named Examples - Specific figures are used throughout: "over 50 million users," "51 billion per day" of telemetry, and "10 times improvements in our builds." - A concrete eval anecdote is given: "The same five character file took the most expensive model 70x more tokens." ### Lessons Framed as Transferable - The speaker repeatedly generalizes from VS Code to other teams: "think how you can encode your expert knowledge," and "try these things, find out where your bottlenecks are." ### Candid Acknowledgment of Failure - Mistakes and limits are acknowledged: "the agent will make mistakes," "we were not convinced yet" about the GitHub Copilot review, and "We should have probably done it earlier." ### Conversational, Informal Delivery - The transcript includes casual asides and audience interaction, such as "Why did everybody leave?" and "Yeah, cool. Kind of similar. Awesome." ### Caveats on Attribution - Some claims are hedged: "I believe we really needed," and "I see a lot of teams..." These are presented as the speaker's perception. --- ## 5. Claims the Material Does Not Independently Demonstrate - **Metric definitions and figures** (for example, the 55% to 86% code survival rate) are presented as internal data. The transcript does not show the underlying data. - **Model names** are rendered inconsistently in the transcript (for example, "GBD 4.1" and "cloud opus 4.6"). The intended names are not verified here. - **Vendor-neutral framing** is not fully established. The speaker is affiliated with VS Code, and promotes its tooling, while also mentioning GitHub Copilot, Xcode MCP, and Playwright. - **Speaker motives** are not stated and are not inferred. --- ## 6. Summary - **Self-described role:** VS Code team member and PM, who also works on iOS apps. - **Audience:** Engineers and engineering leaders seeking process change, primarily at a conference. - **Claimed scope:** How agents are used to build VS Code itself, with emphasis on feedback loops, quality gates, and faster learning. - **Presentation style:** Metric-driven, candid about failures, conversational, and framed as transferable practice. # Delivery Observation: Harald Kirschner — "How VS Code Went from Monthly to Weekly Releases with AI" ## Overall Delivery Style The delivery is **conversational, fast-paced, and demo/story-driven**. The speaker talks through slides casually, with frequent asides and self-interruptions ("So, it's not about how we build agents… but this is about how we use agents"), which suggests a talk being re-delivered from memory rather than read. ## Recurring Linguistic Patterns ### "Next up" as a structural hinge The most consistent verbal signpost, used repeatedly to move between slides/topics: > "Next up you have skills." > "Next up, when agents actually work on UI…" > "Next up, we have the self-correcting loop…" > "Next up, we have prototyping…" This is a stable, deliberate teaching habit — the audience can reliably hear it at nearly every section boundary. ### "So" as a connector and opener Extremely frequent — opening sentences and linking clauses throughout: > "So, it's not about how we build agents into VS Code…" > "So let's start shipping faster." > "So one of the examples is our VSC bench." ### Triadic framing: "ship faster, ship at quality, learn faster" The talk's three-part structure is verbally reinforced at multiple points: > "shipping code faster and that's the easy part" > "Okay, now we're holding we're shipping faster. We got to hold quality." > "to actually ship higher quality, ship faster and learn faster" This repetition appears to be an intentional rhetorical scaffold, not a slip. ## Teaching Habits - **Concrete numbers as evidence**: nearly every claim is paired with a figure — "GBD 4.1 started at 55%… cloud opus 4.6 is now today at 86%", "10 times improvements in our builds", "51 billion per day", "70x more tokens", "over 50 million users". - **Anchoring abstractions to a named product feature**: abstract advice is always followed by a concrete VS Code example — e.g., "creating agents.mmd" for codebase readiness; "one skill we added early on was infusing all the accessibility best practices" for skills. - **Audience interaction**: live hand-raising checks — "Who has filed a GitHub issue? Who has had problems in VS Code before? Yeah, cool." and comparing to a prior session: "This the last crowd had less hands up." - **Self-deprecating and playful openers**: "Why did everybody leave? No." / "kudos to you for catching this one." - **Personal participation as proof**: "can I as a PM effectively VIP code in the VS code repo which I do um" — using his own behavior as the litmus test. ## Vocabulary & Word Choice - **Repeated key terms**: "feedback loop" (many instances: "get this feedback loop of is it all working", "a really tight feedback loop", "we need that feedback loop when it makes mistakes"), "bottleneck" ("find your bottlenecks", "these any slow part of your CI/CD will compound", "find fix that next bottleneck"), "agents", "harness", "signal to noise". - **Casual register / slang**: "vibe" ("Now that everybody can vibe"), "token maxing", "yolo releases", "scrappy PR", "These moments are magical". - **Filled pauses**: frequent "uh", "um", and mid-sentence restarts ("how do you build these loops? And then figure out after you have these these success moments"). ## Stable Patterns vs. Isolated Moments **Stable (high confidence):** - "Next up" section transitions (5+ occurrences) - Number-backed claims throughout - The three-phase "faster → quality → learning" structure stated multiple times - Concrete example following every recommendation **Isolated (low confidence, single occurrence):** - The playful opener about the rerun talk - The TypeScript Go aside and the "hello world" eval anecdote — these fit the evidence-heavy pattern but appear only once each ## Uncertainty Notes - The transcript appears auto-generated ("GBD 4.1", "VIP code", "multi- aent system"), so exact spellings of product/model names cannot be verified from the transcript alone. - Tone is assessed only from text; no inference is made about the speaker's personality or motivations beyond observable word choice. # Observation Record: "How VS Code Went from Monthly to Weekly Releases with AI" ## Source and Attribution - **Video ID:** 01a15ca7-1df6-7458-b24f-eac82f5f11c2 - **Speaker:** Harald Kirschner, per the video title. The transcript itself does not name the speaker. It describes the speaker as working on VS Code and as a PM who contributes to the VS Code repo. - **Speaker's role as stated:** Works on VS Code; also works on iOS apps (stated as "on my side"). - **Nature of content:** A conference talk describing the speaker's account of team practices. Claims about outcomes are the speaker's reported experience and are not independently verified in the transcript. - **Transcript notes:** The opening refers to this as a "rerun of a talk that she gave yesterday." The pronoun does not match the title's speaker reference, and the transcript contains some automatic-transcription artifacts (e.g., "VIP code," "Xcode MCP," ">>"). Where meaning was unclear, it is noted below rather than resolved. ## Core Thesis (as stated by the speaker) > "It's about evolving the whole system, how you're shipping software to evolve to actually make better use of AI along the whole process." The speaker explicitly distinguishes this from using AI more frequently day-to-day: > "It's not about using more AI as you work day-to-day, which a lot of companies are pushing. Like you're not token maxing." The stated goal is to ship **more effectively**, not only faster. ## Reported Metrics | Metric | Reported Value | Context stated | |---|---|---| | Agent code survival rate | GPT-4.1: 55%; Claude Opus 4.6: 86% (current) | Share of agent-written code that gets committed; improvement attributed to harness and models | | Release cadence | Monthly for ~10 years → weekly | Since VS Code 1.0 | | Build time improvement | ~10x | Attributed to TypeScript Go | | Telemetry volume | ~51 billion events/day | Raw telemetry before filtering | | Token cost variance | 70x more tokens for the most expensive model | Simple "hello world" file eval; speaker says it was not the high-reasoning model | | User base | 50+ million | Described as a "very small team" | The speaker presents the survival rate as evidence of "developer trust and confidence," but this is an interpretation of the metric, not a measured finding about trust. ## Position Areas ### 1. Shipping Faster Creates New Bottlenecks - Faster shipping shifts the problem to **code review** and to **shipping high-quality code that does not constantly break**. - Because VS Code ships binaries to user machines, the speaker states that "recovery is even more costly." - Speed of learning is framed as a larger challenge than speed of code generation. > "Shipping code faster and that's the easy part." ### 2. Codebase Readiness - **Agent documentation:** The speaker recommends `agents.md` files that are "lightweight enough" and give agents a map of the codebase. - These are described as **living documents** that should be iterated as agents make mistakes. - The speaker notes that teams with strong developer-experience investment benefit because good onboarding documentation helps agents as well as humans. ### 3. Encoding Expertise in Skills - An **accessibility skill** was created so that practices previously held by one person are available to all contributors. - The skill is reviewed and maintained by the accessibility area owner. - The general principle: encode expert knowledge into skills to unblock users who are frequently pulled in for input. **Litmus test stated:** Whether a PM can effectively contribute code to the VS Code repo, and how engineering accepts that work, including whether it works out of the box, locally, or breaks after shipping. ### 4. Build Performance - The migration to **TypeScript Go** is described as "a massive one." - The speaker argues that slow CI/CD matters more with agents, since "10 agents, 20 agents" hit the same bottleneck and slow stages "compound." ### 5. Verifying UI with Agents - The speaker states that agents often produce UI that "looks perfect" but is misaligned, "even with the latest models." - Central principle: > "If your agent cannot use your application, your product directly to get this feedback loop of is it all working, then that's a really big investment that pays off every time you work on UI." **Two investments described:** - **Component browser:** An automated build that screenshots every component on change and flags differences, including unintended ripple effects. Used to speed review. Developers are asked to attach a video or screenshot to PRs. - **Self-correcting loop:** A `/launch` skill uses Playwright to run VS Code through scenarios, diagnose issues, and validate fixes. The speaker notes that VS Code is "all HTML" running in Electron, which makes Playwright suitable. The speaker names Xcode MCP as a favorite for iOS work and says other tools exist for Android. ### 6. Code Review - The speaker reports that the GitHub automated reviewer was initially not trusted: > "We're not convinced yet but over the next months and weeks they have massively improved." - Automated review is now **mandatory** for every PR. - Review effort is planned to scale with repository risk (low, medium, high). - Humans do not review a PR until automated comments are resolved. ### 7. Issue Triage and Automated Fixing - **Signal and noise:** AI now filters spam, enriches, translates, and assigns area owners to issues. The speaker reports that AI-assisted filing also yields "better issues" from users with different language backgrounds, while acknowledging "a lot of issues." - **Community contributions:** The speaker says that, contrary to an assumption that AI produces many low-quality PRs, the number of community-merged PRs is rising: > "Which is really amazing that we can unlock more community contributions as well." - **Human feedback layer:** A Chrome extension lets humans correct agent triage mistakes (e.g., duplicates). The speaker stresses feeding this back into agent work. - **Error-stack pipeline:** Uses internal telemetry, not external tools. The pipeline filters raw telemetry to error stacks with full stacks, groups them via fingerprinting, and combines this with an existing ML sorting layer. An issue is filed and a PR is auto-created with diagnosis. The speaker gives one example in which the agent found a missing cancellation request in the RPC protocol. - **Human oversight:** Fixes land "mostly automatically with some human oversight to still approve the PR." *Transcription note:* The speaker says "we have 10 issues filed assigned to specific area owners." The surrounding sentence is garbled and the meaning of "10" (per day? per batch? an example count?) is not recoverable from the transcript. ### 8. Release Safety - Previously: "yolo releases," with testing followed by a full 100% rollout. - Now: **staged rollouts**, with monitoring of error stacks and issues during rollout. - Justification: rollbacks on installed applications are expensive. > "We should have probably done it earlier." ### 9. Learning Faster: Evals and Experimentation - **VSC bench:** An internal eval suite. New scenarios are filed as GitHub issues, and an agent bootstraps the scenario from a template. - The process: hill-climb against evals offline, then confirm with online experimentation. - **Eval insight:** A "hello world" file-writing scenario showed large token variance across models. The speaker says this is why "understanding your evals is critical." The speaker explicitly declines to discuss specific models, focusing instead on how models react to a given harness. ### 10. Prototyping and Team Structure - Prototypes are used as conversation starters rather than as PRs to land. - Feedback cycles have shortened from **monthly release to weekly to daily** sprints. - Smaller squads and durable ownership are described as ways to move faster. **Example:** An "ask questions" tool started as a "very scrappy" PR. Within weeks it had "a perfectly polished and designed experience," and the speaker says the early PR unblocked participation by giving the team a shared idea. ## Qualifications and Tensions Preserved - **Trust vs. verification:** The speaker reports rising code survival (55% → 86%) as a signal of trust, but also says agent UI output often looks correct and is not, and that automated review was initially unconvincing. - **Volume vs. quality:** Issue and PR volume have increased. The speaker says quality issues rose, while also noting better-quality issues and rising community merges. - **Automation vs. oversight:** Fixes are described as landing "mostly automatically," yet humans still approve PRs, and the feedback layer exists because "the agent will make mistakes." - **Speed vs. risk:** Stated concern that agents "compound" bottlenecks, while staged rollouts are introduced to "derisk" a process that previously shipped to 100% at once. - **Model choice:** The speaker explicitly says they "don't actually talk about the models" in this context, focusing on harness–model interaction. - **Cost:** Review effort is to be tuned by risk for "cost benefit." No cost figures were given. ## Unresolved Questions and Gaps - **Generalizability:** The speaker's experience is with a very large open-source project and a "very small team." The transcript does not address how applicable these practices are to smaller or less-instrumented projects. - **Metric definitions:** "Code survival" is defined as the share of agent-written code committed. The transcript does not describe how agent-written code is identified, nor how "trust" is measured beyond this proxy. - **Causality:** Improvements (survival rate, build speed, release cadence) are described alongside several changes at once (new models, harness improvements, TypeScript Go, skills). The transcript does not isolate individual contributions. - **Quality outcomes:** The speaker reports the tooling and shipping cadence but does not report defect rates, rollback frequency, or user-impact measures after the weekly cadence began. - **Garbled passages:** Several sentences are unclear in the transcript (e.g., the "10 issues" line, "VIP code," "one of the biggest or the biggest open-source project"). These are not resolved here. ## Summary of Stated Positions - **Core claim:** Agent adoption requires changes to the whole delivery system, not just more AI use. - **Central principle:** Agents need feedback loops, including access to the running application, fast CI, and automated review. - **Quality stance:** Gate on automated review, component screenshots, staged rollouts, and human approval. - **Learning stance:** Maintain evals, use prototypes for fast feedback, and keep a human feedback layer on agent output. - **Stated bottleneck framing:** Each improvement reveals the next bottleneck, so the process is described as "a constant iterative cycle." # Video Attribution & Provenance Record **Video:** How VS Code Went from Monthly to Weekly Releases with AI — Harald Kirschner **Video ID:** 01a15ca7-1df6-7458-b24f-eac82f5f11c2 --- ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Harald Kirschner** (implied by title) | **Creator / Presenter** — VS Code team member, conference speaker | Title names him; transcript is first-person talk: "I'm working on VS Code" | | Unidentified prior speaker | Referenced speaker — original talk given "yesterday upstairs"; this is a rerun | "This is the rerun of a talk that she gave yesterday upstairs" (gender pronoun inconsistent with title attribution — **uncertain**) | | Audience | Non-speaking; applause only at end | "\[applause\]" marker | **Attribution uncertainty:** The title credits Harald Kirschner, but the transcript opens referencing "a talk that she gave yesterday" — the presenter's identity is inferred from the title, not explicitly stated in the transcript. No curated metadata beyond the title was supplied. --- ## Referenced Sources (cited within talk) - **GitHub** — VS Code described as "one of the biggest or the biggest open-source project on GitHub" - **VS Code blog** — eval experiment ("you can read about on our blog") - **GitHub Copilot automated code review** — third-party product, now mandatory on PRs - **Models referenced (by name, not evaluated in detail):** "GBD 4.1" (55% code survival) → "cloud opus 4.6" (86% code survival) — *model names as transcribed; possibly transcription variants of GPT/Claude model names (uncertain)* - **Tooling referenced:** TypeScript Go, Playwright, Xcode MCP, MCP plug-in/skills system, agents.md files, Chrome extension (in-house), VSC bench (in-house evals) ## Third-Party Claims - Code survival metrics: GBD 4.1 at **55%**, cloud opus 4.6 at **86%** — presented as VS Code internal measurement - TypeScript Go delivered a **10x build improvement** — team's internal claim - Eval experiment: same 5-character file took the most expensive model **70x more tokens** — readers directed to the blog for details - Community merged PRs increasing, not just low-quality PRs ## Internal (First-Party) Claims & Systems Described - VS Code ships to **50M+ users** with a small team - Monthly → weekly release cycle (~10 years of monthly releases prior) - Telemetry: ~**51 billion events/day**, filtered to full error stacks, auto-grouped → issues + auto-opened PRs - Stage rollouts replacing previous "yolo releases" - Component browser/browser automation and `/launch` self-correcting loop - AI issue triage (filtering, enrichment, translation, area-owner assignment) with a human feedback layer ## Promotional Segments - **Booth invitation (explicit):** "You can come by the booth and see us showing of agents" and closing "Come by code booth. Happy to chat now." - No third-party sponsorship or paid promotion is evident. ## Availability / Access Limits - **No curated metadata supplied** beyond the video title — no speaker bio, publication date, event name, or venue confirmed (conference setting inferred from audience interaction and applause). - **No comments section information** available from the supplied material. - Model names in eval metrics are **not detailed in the talk itself**; the speaker states "we actually don't talk about the models" — details deferred to the blog. - The referenced blog post, GitHub issues, and component screenshots are described but not directly accessible from the transcript. Hey uh everyone. This is a hard tone to follow. Why did everybody leave? No. Um this is the rerun of a talk that she gave yesterday upstairs. So this will be better than the one yesterday. So uh kudos to you for catching this one. I'm working on VS Code and we move from a monthly release to a weekly release cycle thanks to AI. And this is about the things we learned, the systems that made this work, not just to ship faster, but actually to ship more effectively. So, it's not about how we build agents into VS Code, which we can try out anytime. You can come by the booth and see us showing of agents, but this is about how we use agents to build the product itself. And that's VS Code, that's the copci and other areas, but also how as you use agents and more AI, things are breaking and you have to rethink actually how you work. And that's a very small team shipping to over 50 million users. What happened is over the past year is we started tracking already early on this code survival metrics in VS code. So that means the percent of agent written code that actually gets committed. how much kind of garbage has the agent produced that you threw out the window again or didn't trust and that you that you read it u and and scrapped. So GBD 4.1 started at 55%. But then over time and over improving the harness and new models cloud opus 4.6 is now today at 86%. And that increase clearly shows developer trust and confidence in shipping AI code faster and more AI code generated. And this success created problems on the VS code side. We're one of the biggest or the biggest open-source project on GitHub. So we saw a massive increase in issues because AI helps to file issues. So we actually see an increase in quality issues but also the counterpart of more automated lowquality issues. We also see more open PRs by the team by our own velocity and by our community. And interesting part we actually see everybody would assume we see a lot of garbage PRs but actually the number of community merged PRs is going up as well which is really amazing that we can unlock more community contributions as well. And as part of this new workflow, we actually moved several months ago, we moved from monthly releases, which happened over 10 years. Since the inception of VS Code 1.0, we have been shipping a new version of VS Code on a monthly basis. Including fun facts like the January release actually ships in February. So when you read the release notes in VS Code, you're wondering why it's February and you're reading the January release notes if you're a month late. No, that's just how we tracked our iterations. So that cadence was to match to velocity, but it also showed us that the it's not about using more AI as you work dayto-day, which a lot of companies are pushing. Like you're not token maxing. You're not spending every day using AI. It's about evolving the whole system, how you're shipping software to evolve to actually make better use of AI along the whole process. And that what makes this velocity actually work and doesn't just generate more code. I'm breaking this down into this first part which a lot of people have been experiencing. I see shipping code faster and that's the easy part. You have like this 100x engineers that has built this beautiful skill MCP plug-in system and they're just shipping code like crazy and slowly maybe more in the more people in the team uh adopt the same patterns and you see a lot more code shipping faster. But the problem is once you ship faster, you get into code review. You get into how do you actually ship high quality code that doesn't constantly break. And that's important for us as VS Code. We're shipping binaries to user machines. So if something breaks, that recovery is even more costly. But once you hit that velocity with shipping high quality code faster with confidence, how do you actually know you're shipping the right thing or if we should ship this thing? And that's where AI kicks in to actually learn faster in the end and shipping better products which a lot of people are missing out as as they write more code is applying that product taste and applying that learning. So let's start shipping faster. Now that everybody can vibe, we can really all ship faster. And it's a big part is getting your codebasation ready. So that's kind of the no-brainer slide that everybody should already been doing is thinking about how to get your code bases agent ready is creating agents.mmd that give it that are lightweight enough and give the agents a good kind of map of the codebase where to look at mapping out that is every agent now at your uh at your fingertips has a slash in it that gives you a good draft and then you keep itering those are living documents that should evolve as the agent makes mistakes and works. And that's where I see a lot of teams that have invested in developer experience actually gain a lot because once they have good documentation and onboarding for a codebase that is read by the agent as well and that can help the agent as much. Next up you have skills. VS Code has always messed heavily in accessibility. So one skill we added early on was infusing all the accessibility best practices and how we think about accessibility into a skill that everybody will use and that previously was one person that had to then got get pulled in or give feedback and now that skill was reviewed and is maintained by the kind of area owner of accessibility. So to bring that that accessibility that short work everywhere into everybody's hands. So think how you can encode your expert knowledge to unblock this like few users that might be always asked for this input into a skill. And lastly once you do it right for us in VS Code the litmus test is can I as a PM effectively VIP code in the VS code repo which I do um and how is that accepted by engineering? How much work does it cost? Is it out of the box working? Is it for me working locally? Or is it eventually breaking when I ship it? So, a lot of these quality improvements will show in the end that you can allow everybody to be more effective in the repo. Another improvement, and that's a massive one, is we switched to TypeScript Go. And if you've been in any developer codebase that takes some time to build or run linting or run CI/CD, you've noticed it's already annoying. But developers find other ways. They can review a PR. They can context switch into something else while runs and then do something else. But once you hit that with agents, you have 10 agents, 20 agents running and hitting the same bottleneck, these any slow part of your CI/CD will compound. And for us, Typescript Go was a 10 times improvements in our builds, which at the scale of automatic PRs running with agents and them fixing getting needing that CI/CD loop for feedback to improve the code is a massive improvement. And that leads to me uh filing big PRs. That was beginning of the year to get the ask questions tool in which I believe we really needed and then bringing it to the team to show to let them try it out in the product. Within a few weeks we had a perfectly polished and designed experience based on my initially very scrappy PR. But it unblocked that this these discussions the participation of everybody else because we had the ground set laid and the idea and the experience was clear. Next up, when agents actually work on UI, and we all had this, that the agent gives you a UI and it says it looks perfect, then you open it and it's all misaligned. Even with SVG, even with the latest models, you still get it. And it's like the number one thing I keep telling everybody now these days. If your agent cannot use your application, your product directly to get this feedback loop of is it all working, then that's a really big investment that pays off every time you work on UI. And there's two areas we invested in. One is the component browser, which is an automated build that runs every time VS Code changes something. Let me open this in this issue as an example. In this case, we added a back button to our customization screen. And because it's changing wide, we actually have this automated process in the back that takes screenshots of every component in VS Code and points out the differences. It allows us to detect unexpected changes like we're changing one component as the ripple effect of something else moving or some other icon disappearing. and allows you to quickly review a PR as well because you also get directly a view and what the PR is actually doing and that's what we ask every developer before we ask them to at least attach a video or screenshot on what they have been implementing so we don't have to run everything from scratch and we can quickly assess and and give feedback and this is now automated with the component explorer next up we have the self-correcting loop because in VS Code we run uh a web browser and open a website. Basically, that's what VS Code is. It's all HTML. We can actually use Playright to automate a lot in the browser. So, we have a /launch skill that would just open up VS Code and give it a specific scenario to click through in VS Code. So, it's a really powerful way to diagnose issues because you can also get the locks along the way, but also validate any fixes you did before and after without you having to click through it. So you can just hit launch move on to the next task and then come back once the agent verified it's worked because it can access your application and to call out this is nice because we have playright and we are a web app uh running in electron but there's many other solutions out there one of my favorites is Xcode MCP because on my site I work on iOS apps and once you see Xcode MCP just creating in VS code without Xcode involvement just making an iOS app opening it up clicking through it, taking screenshots. These moments are magical and give you a really tight feedback loop as well. So there's other tools available for Android as well to make this work. Let's talk about code review, which I think is the next big investment. We know in the past when we enabled code we on GitHub the automated GitHub copiloted one we're not convinced yet but over the next months and weeks they have massively improved and now it's a mandatory uh review every time somebody opens a PR and there's more improvements coming actually dial in the uh the effort that a code review does depending on how much risk you have in this repo so you have a low medium high effort for code review so you can dial in kind of the cost benefit as well but for us is if a review after review is done, humans will not even look at PR until all the comments are resolved and and addressed. Okay, now we're holding we're shipping faster. We got to hold quality. And a big signal for us because we're on GitHub is of course issues. Uh who has filed a GitHub issue? Who has had problems in VS Code before? Yeah, cool. Kind of similar. Awesome. This the last crowd had less hands up. So it's good. Um, yeah. So it's a massive, it's a really good signal. And as I said, we get sometimes even better issues now because of AI and more people with different language backgrounds being able to file very good issues. But we also get a lot of issues. And previously engineers actually triage them manually. And now we have AI doing the filtering to get out spam enriching and translating in the end assigning area owners for all the issues to raise the signal to noise quality of this. We also add a human layer to that once you are in issue because the agent will make mistakes and we need that feedback loop when it makes mistakes. We create a Chrome extension that allows you to fix duplicated issues as well. So we have the agent doing the work and we have a feedback loop where humans can still fix it that we can bring back into the into the agent work up front. That's a really important thing to think about as you have agents doing the work that you get the human feedback as well. The other input we have is a lot of error stacks. Every time there's an exception and most apps will have this you get an error stack reported and we actually don't use any external tool for that. It's all in our own data and we built all the tooling around it and it's mostly we already had it. We already had an ML doing sorting on top and this is now actually done by a mix of criteria. So one is we collect the raw telemetry. We filter it down and that's like 51 billion per day. We filter it down to just error stacks that actually have the full stack. Then we group it and bucket it. That's more of the fingerprinting exercise. And then in the end after we all of that we have 10 issues filed assigned to specific area owners that a PR is autocreated to actually already try to fix the problem. This is our error page where we see all the stacks including how many hits and how many users affected so we can quickly react to anything that's coming. And once we have this working we get a issue filed and a PR automatically opened. This is the issue. is the error with some initial diagnosis and then the PR got opened with and the agent actually goes in and finds out why does this happen who was responsible in this case um added a change that was connected to this to find f find symbols and then it figured that the on cancellation request wasn't in the RPC protocol so the agent already opened the PR we could just merge it and that's a really great automated way you don't care about errors you just want to get a more stable code base. So all of this automated the triage initially mix of agents and deterministic systems then the issue is filed handed off to an agent in the PR multi- aent system and in the end we have fixing fixes landing mostly automatically with some human oversight to still approve the PR. There you go. So previously VS Code did yolo releases in a way that it was release day everything was stable. We did our testing and we shipped the release to 100%. We just opened the floodgates but because of agents we want to derisk that process. We should have probably done it earlier. So now actually we do stage rollouts and as we do the roll out we monitor error locks and issues and everything else. So just like good citizens in a in an web application, we now apply the same because rollbacks on an install app are so expensive. Okay, now we have this shipping faster, shipping at quality. How do we actually do this in a in a speed that we actually learn faster, which is the which is a bigger challenge with AI? You actually want to build a better product faster. So one of the examples is our VSC bench. We're shipping an agentic product. So it's important that we have our own evals of our product always at hand and being really easy to extend. It's all in the GitHub issue. It's all in GitHub. When we file an issue to add a new scenario, it's actually an agent picking up from a template and bootstrapping the scenario for you. So we make it as easy as possible to bring in all the developer scenarios that are coming up in issues through customer conversations through all these channels to bring them back into our emails. So we then start hill climbing actually against them and then com also confirm that it's all improving offline and then an online experimentation. So classic product life cycle for um for us and we see these interesting things in our stack. Understanding your evals is critical. One experiment we did that you can read about on our blog is that we gave a very simple eval scenario of just write a file with hello world just testing out our eval harness end to end and how different models react. The same five character file took the most expensive model 70x more tokens and that was not the high reasoning model. You can read more about unblock. We actually don't talk about the models, but just that insight in how your models react with your harness is extremely is a very important exercise. Next up, we have prototyping, which I do a lot of because I noticed that I don't actually want to land most of my work into VS Code PRs itself. It's a great way to start a conversation, but most often I want to have a quick conversation because now we're meeting on a daily basis where we just want to have very quick feedback loops on here's the idea, here's how it could look like, what if we do this. Next day you come back with updated uh prototypes and you just keep having this conversation. So much quicker feedback from a month release cycle to weekly to daily sprints where you just keep working on problems and prototypes unlock those deeper discussions and how the experience should look like. It also means that we have the durable ownership in internal work as I mentioned daily sprints um smaller squads, smaller work streams that really prioritize and drive smaller sets um of areas to move faster. So try these things, find out where your bottlenecks are to actually ship higher quality, ship faster and learn faster and find fix that next bottleneck. And don't just tune how you work with agents, but tune where how how do they get feedback? How do you build these loops? And then figure out after you have these these success moments, how can you improve that feedback loop for the next moment of what's blocking you to move even faster and it's a constant iterative cycle across all these cycle ones. That's it. Come by code booth. Happy to chat now. Otherwise, happy coding. \>\> \[applause\]