Talk by Eugene Yan (Anthropic) on using large language models to secure source code. Cites examples such as Mozilla's reported 20x jump in monthly security fixes in April 2025 and Anthropic's scan of over a thousand open source repos, arguing that finding bugs is no longer the hard part and the bottleneck has shifted to verification, triage, and patching. Presents a six-step workflow — threat model, sandbox, discovery, independent adversarial verification, triage, and patching — demonstrated through a running example of a SQL injection in a Python order lookup service. Closes with advice on starting with open source dependencies and keeping human oversight before automating. # Video Identity Observations: "Using LLMs to Secure Source Code — Eugene Yan, Anthropic" ## Speaker & Affiliation - The speaker introduces himself directly: **"Hi, I'm Eugene, [a] member of technical staff at Anthropic"** (transcript renders it "Entropic"). - He describes his role scope: **"I've spent the last several months working with security teams and Claude to find and fix vulnerabilities in code and systems."** - The title frames this as an Anthropic-affiliated talk. ## Explicit Self-Description / Claims About the Work - He claims institutional experience with **"dozens of organizations"**: "lessons we've learned from working with dozens of organizations to improve their security posture." - He presents first-party Anthropic results: **"This is Anthropic sharing of our own work scanning more than a thousand open source repos. From 23,000 candidates, 6,200 of them were rated as high or critical."** - He shares an opinion/position: **"finding vulnerabilities now is quite straightforward. The bottleneck has now shifted to verification, triage, and patching."** ## Intended Audience (Demonstrated) - The talk is delivered live to a mixed engineering audience; the speaker polls the room: - **"How many of you here are security engineers in your day job?"** - **"How many of you here are engineers in your day job?"** - He explicitly targets **practitioners rather than specialists only**: "I want to get this message out to as many people as I can," and jokes, "Security engineers in the room, please don't boo me if I oversimplify things." - Content level is practical/beginner-to-intermediate: he asks **"How many of you here know what a threat model is?"** and defines it. - The closing call to action addresses hands-on engineers: **"start interactively... hands on the wheel with Claude Code or your favorite IDE."** ## Claimed Scope of Content - A structured agenda is stated explicitly: 1. **Three high-level trends** in model capabilities, vulnerability counts, and shifting bottlenecks. 2. **Distilled lessons** from working with dozens of organizations. 3. **How to get started "this week"** with links to resources. - The technical core is a **six-step process** ("most teams converge in approximately these six steps"): threat model, sandbox, discovery, verification, triage, patching — presented via a running **"order service"** example. ## Demonstrated Presentation Patterns - **Evidence-heavy style**: cites external sources by name (UK AI Security Institute benchmarks, Mozilla Firefox bug-fix statistics, Log4Shell, Heartbleed) and quotes a CISO: *"The model has great context of the code but poor context of the system,"* and a partner team: *"the biggest lever we had is having the model test beds."* - **Interactive delivery**: repeated audience polls and prompts to shout out answers ("Can anyone here spot the vulnerability? You can shout it out."). - **Product/resource promotion**: ends by pointing to Anthropic resources — "cloud security to scan your codebase," a blog, open source repos with "interactive skills and autonomous harnesses," and a getting-started sequence. - **Prescriptive takeaway framing**: "if you only remember one thing — well, here are the three things," listing: start now with open source dependencies, climb the learning curve interactively, and aim beyond scanning at verification/triage/patching. ## Distinguishable: Claims vs. Demonstration | Type | Evidence | |---|---| | **Creator claim** | Anthropic's scan results (6,200 high/critical of 23,000 candidates); "dozens of organizations" experience | | **Creator claim** | "Current models are very good at spotting simple stuff like this" (SQL injection demo) | | **Demonstrated** | Educational, tutorial-style structure with worked example and audience interaction | | **Demonstrated** | Audience is practicing engineers/security engineers at a conference-style event | # Delivery Observations: "Using LLMs to Secure Source Code — Eugene Yan, Anthropic" ## Tone and Register - **Conversational and informal** throughout. The talk opens with hedged, filler-laden speech: *"Hi, I'm Eugene U member of technical staff at Entropic. Um, I've spent the last several months with working with security teams..."* - **Self-deprecating and audience-deferential**: *"security engineers in the room. Please don't boo me if I oversimplify things."* - **Direct, plainspoken framing of stakes**: *"Things that can be solved with money are not really problems. But human attention doesn't scale."* ## Recurring Phrases and Verbal Habits - **"You know"** appears constantly as a connective filler: *"You know these tasks require identifying exploiting security weaknesses"*, *"you know, the key asset is the data you know contains customer PII."* - **"Right?"** as a rhythm marker: *"three highle trends... right?"*, *"this is the generative verifier loop... right? You're building loops."* - **"So"** begins a large share of sentences: *"So why do we care about so much about cyber security now?"*, *"So we have them both working separately."* - **Um/uh fillers** and stumbles are frequent: *"truly optim uh truly exploitable"*, *"we can can can run everything."* ## Audience Interaction Style A distinct, repeated teaching habit — **live polling via "hands up" / show of hands**, used at least four times: > *"quick show of hands. How many of here here are security engineers in your day job?"* > *"Hands up. Who here remembers lock for shell?"* > *"How many of you here know what a threat model is? Amazing. I love that."* > *"Can anyone here spot the vulnerability? You can shout it out you guys."* He **adapts to audience response**: *"Okay, great. I have better understanding of how to give this talk now."* ## Structural / Teaching Patterns - **Explicit roadmap up front**: *"first I want to share with you three highle trends... Then I want to distill the lessons... And finally, I want to share how you can get started this week."* - **Running concrete example** ("order service") carried through every step: *"Now to make things concrete I want to share with you how they apply on a hypothetical system."* - **Analogies to ML concepts** for the engineering-heavy audience: *"discovery optimizes for recall... Verification optimizes for precision"*, *"it's a regular machine learning pipeline."* - **Recap and "remember three things" close**: *"Okay, if you only remember one thing well here are the three things you want I want you to remember."* - **Quoting external sources verbatim**, flagged as such: *"I'm going to read off the screen here. Early experiments showed that some promise..."* and a quoted CISO: *"the model has great context of the code but poor context of the system."* ## Vocabulary and Framing Choices - Consistent use of practitioner jargon: **PoC/"pox"** (with a definition offered: *"I'm going to call them pox to confirm that they are true positives"*), true/false positive rate, threat model, sandbox, detonate. - Vivid action verbs: *"deliberately detonate proof of concepts"*, *"run and detonate the pox"*, *"a fresh discovery agent try to attack the patch code again."* - Quantified claims as evidence style: *"20x of last year's average"*, *"true positive rate was almost 100%"*, *"cut my prompt size by maybe about 50%."* ## Personal-Anecdote Insertions He occasionally anchors advice in his own experience: > *"That's what I found with every new model version of StepJum, I actually have to cut my prompt size by maybe about 50%."* > *"This is code that I would have written right as me as a non security engineer."* ## Uncertainty Notes - The transcript appears to be **auto-generated with errors** ("Entropic" for Anthropic, "lock for shell" for Log4Shell, "track/threat model" confusion, "claw" for Claude). Quotations above reproduce the transcript as given; some words may not reflect the speaker's actual delivery. - Patterns like "you know" and "so" are clearly stable; tone judgments rest on a single talk, so consistency across other talks is unverified. # Position Extraction: Using LLMs to Secure Source Code — Eugene Yan, Anthropic ## Speaker - **Eugene Yan**, Member of Technical Staff at Anthropic (per curated metadata and transcript self-identification), describing work with security teams using Claude to find and fix vulnerabilities. ## Core Claim > "Finding vulnerabilities now is quite straightforward. The bottleneck has now shifted to verification, triage, and patching." ## Evidence Cited ### Rising model capability - **Time-horizon benchmarks**: The UK AI Security Institute's cybersecurity variant measures model task duration vs. humans; Yan observes a **step jump above the previous regression line**, indicating a step change in capability. - **Mozilla Firefox bug-fix volume**: average ~20/month in 2025; ~60–70 in Feb/March; ~400 in April (~20x last year's average). Mozilla attributed ~two-thirds to Claude preview (~271 fixes). - **Anthropic's own scanning**: 1,000+ open-source repos, 23,000 candidates, 6,200 rated high/critical; 1,600 reported to maintainers, ~100 patched upstream (at time of update). ### Agentic harnesses - Quoting Mozilla (pre-Claude-preview): early experiments showed promise but **high false-positive rates** made scaling impractical; **agentic harnesses** that reliably detect issues changed this — "the harness and the model working together can get a lot of work done." ## The Six-Step Process ### Setup (per codebase) 1. **Threat model** — a document providing system context the model lacks. - Claim: teams found a well-documented threat model raises **true positive rate to ~90%**; Yan considers >75% great. - Quoted CISO: "The model has great context of the code but poor context of the system." - Build via bootstrapping (give model docs, commits, past CVEs to infer unpatched ones) then having the model **interview the system expert**. 2. **Sandbox** — isolation (VM without egress or cloud credentials, to prevent data exfiltration or production drops) and **reproducibility** (consistent baseline containers). - Quoted team: "The biggest lever we had is having the model test beds… where they can run and detonate the PoCs to confirm that they are true positives." ### Loop 3. **Discovery** (optimizes **recall**): - **Context engineering**: more shared context improves coverage and reduces false positives. - **Simpler prompts**: Yan states he cuts prompt size ~50% with each model step-up; newer models can infer goals like "look for where untrusted data hits the trust boundary." - **Tools**: a pentesting team gave the model tools (query API, check responses, read logs/source); true positive rate was "almost 100%" because the model could verify in the loop. 4. **Verification** (optimizes **precision**): - Keep discovery and verification **separate**: self-verification can cause self-censoring and hurt recall. - Verification agent should be **independent** (no access to discovery reasoning) and **adversarial** (assume the vulnerability is false), setting a high bar; detonate PoCs in the sandbox to confirm. 5. **Triage**: - Sending product engineers all true vulnerabilities—even medium/low—**loses their trust**; "the scarcest resource now is engineer attention." - Severity depends on magnitude (ring zero vs. ring one) and likelihood; **business context matters** (compensating controls can downgrade; sensitive data can upgrade). Human review example: agent rated SQL injection high; operator downgraded to low likelihood due to an application firewall blocking SQL injection and internal-only, non-internet-facing service. 6. **Patching** — validation ladder: - Original PoC stops working; test suite stays green; a **fresh discovery agent re-attacks the patch**; feedback improves patch quality; human confirms before merge. - Also: document fixes so the loop improves — "operational expense… becomes capital expense. You get better with each iteration." - Framing: the loop "if you squint… looks like an ML pipeline: recall, precision, ranking, and closing the loop." ## Non-Technical Bottlenecks - Quoting a former director: "non-technical problems are an order of magnitude harder than technical problems." - Compute cost is not a real problem ("things that can be solved with money are not really problems"); **human attention doesn't scale**. - **Vulnerability routing**: at hundreds per month, manual curation fails; simple routing to code/service owners may not even need an LLM. - **Severity calibration**: product engineers and security engineers (red/blue team) disagree on severity definitions; requires writing agreed rules down once. - **Patch review**: Yan states he doesn't know many companies doing fully automated patch review for security patches. ## Advice (his three takeaways) 1. **Start now**, with open-source dependencies or whatever is comfortable. 2. **Climb the learning curve** — don't aim for automation immediately; work interactively (Claude Code or favorite IDE) to learn gaps and low-precision areas. 3. **Don't just aim for scanning** — verification, triage, patching, and organizational processes are the real bottlenecks. ## Qualifications & Tensions Preserved - Yan acknowledges oversimplifying for a mixed audience ("security engineers… please don't boo me"). - Claim that "finding vulnerabilities is straightforward" coexists with heavy investment required in threat models, sandboxes, and harnesses. - Figures (Mozilla attribution, true positive rates) are reported secondhand from teams/Mozilla, not independently verified in the talk. - Full automation of patch review is explicitly left unresolved. # Speaker & Provenance Record — Video `01a15ca8-a078-7892-93b6-3c2935b7f64f` ## Video **Title:** Using LLMs to Secure Source Code — Eugene Yan, Anthropic ## Speakers | Speaker | Role | Evidence | |---|---|---| | **Eugene Yan** | **Creator / primary presenter** | Self-introduced: *"Hi, I'm Eugene, member of technical staff at Anthropic"* (transcript renders as "Entropic"); describes months of work with security teams using Claude ("claw"/"Claude Code") to find and fix vulnerabilities. | | **Live audience** | Unnamed participants | Presenter conducts show-of-hands polls (security engineers, engineers); an audience member answers the SQL injection question ("Exactly" — response not captured). | No guest, narrator, or other identified speakers. No curated metadata was supplied. ## Referenced Sources & Third-Party Claims (per speaker) - **UK AI Security Institute** — time-horizon cybersecurity benchmark; models showing step-jump in capability (cited by presenter). - **Mozilla Firefox** — monthly security bug-fix counts: ~20 avg (2025), ~60–70 in Feb–Mar, ~400 in April; presenter states Mozilla attributed ~two-thirds to "Mistral preview" (~271) — third-party attribution, not verified in this record. - **Mozilla (second citation)** — quote read on screen about agentic harnesses reducing false positives. - **Anthropic's own work (presented by presenter)** — scanned 1,000+ open source repos; 23,000 candidates, 6,200 high/critical, 1,600 reported to maintainers, ~100 patched upstream. - **Historical vulnerabilities cited:** Log4Shell (Java logging; Belgium defense ministry breach; fintech leak of 2M users), Heartbleed (OpenSSL). - **Anonymous teams/CISO quotes** — e.g., CISO: *"the model has great context of the code but poor context of the system"*; a team: *"the biggest lever we had is having the model test beds…"* — attributed only to unnamed teams worked with. ## Promotional Segments - **Closing segment:** presenter promotes Anthropic offerings — Claude Code, a blog, open source repos with interactive skills and autonomous harnesses, and a getting-started step list. Flagged as **promotional**. ## Other Provenance Notes - Hypothetical "order service" example (SQL injection, threat model, sandbox, triage, patch) is presenter-constructed illustration, not a real system. - Presenter notes prompt-engineering personal experience ("cut my prompt size by ~50%"). - **Comment availability:** not indicated — no metadata supplied. Hi, I'm Eugene U member of technical staff at Entropic. Um, I've spent the last several months with working with security teams and claw to find and fix vulnerabilities in codes and systems. So, I want to share with you what we've learned as much as we can. Before I get started, quick show of hands. How many of here here are security engineers in your day job? Okay. Okay. How many of you here are engineers in your day job? Oh, sweet. Okay, great. I have better understanding of how to give this talk now. Um, security engineers in the room. Please don't boo me if I oversimplify things. I want to get this message out to as many people as I can. So, first I want to share with you three highle trends that we are seeing in terms of model capabilities, right? Number of vulnerabilities found um and where the bottleneck is shifting to. Then I want to distill the lessons we've learned from working with dozens of uh organizations to improve their security posture as well as the bottleneck that you might encounter and how you can try to address it and prep for it. And finally, I want to share how you can get started this week with links to resources. So why do we care about so much about cyber security now? Um time horizon benchmarks track the length of time an AI model can complete a task measured against an actual human doing the task. Um so we all know about the meter eval task. You know how how much knowledge what a engine can do. The UK AI security institute has a cyber security version of this. You know these tasks require identifying exploiting security weaknesses in target systems and you know testing skills like reverse engineering and web exploitation. So in this chart we see that models are able to increasingly do longer cyber security task. Um but what's also interesting is that you kind of see a step jump against the previous regression line. So these models are a step improvement in capability. And what does this step improvement in capability buy us? Recently Mosilla Firefox published a number of security bug fixes they made each month. The average in 2025 was about 2020 20ish. And then you can see in February and March it kind of 3xed to about 60 and 70. And then in April it 7xed to 400. So what this means is that what's happened in April is 20x of last year's average. Um they attributed about twothirds of this to mess preview about 271 which shows that frontier models can help defenders like yourself find and fix vulnerabilities at scale. We also uh what are some notable vulnerabilities? Hands up. Who here remembers lock for shell? Anyone? Anyone was here over the December holidays like scrambling to patch this seu? So log for shell um is a bug in the Java logging library. If an attacker sends a string, you log it, the attacker can run code on your system. The Belgium defense ministry was breached within days and a fintech platform uh leaked 2 million users data. And then before that we had heartbleleed which is an a bug in open SSL. You know so much of the of the internet runs on OpenSSL and we we all know what impact it had. This is entropic sharing of our own work scanning more than a thousand open source repos. Um from 23,000 candidates uh 6,200 of them were rated as high or critical and at the time of the update 1,600 of them were reported to maintainers and about 100 patch upstream. We shared our observation that finding vulnerabilities now is quite straightforward. The bottleneck has now shifted to verification, triage, and patching. And I want to share with you everything we've learned about that so you can get ahead of it. So how do we do this in our own systems and code bases? Um, two words, agentic harnesses. I'm citing Mozilla again, and this was even before they worked with me preview. I'm going to read off the screen here. Early experiments showed that some promise, but the high rates of false positives made it impractical to scale. But the introduction of agentic harnesses that can reliably detect security issues has changed this. Right? These models and harnesses together can find real bugs and dismiss nonreproducible speculation. So the harness and the model working together can get a lot of work done. We found that most teams converge in approximately these six steps. And a big chunk of my talk is going to be about these six steps. Right? Right. The first two steps are set up, right? You do a bit of upfront investment per codebase. How many of you here know what a threat model is? Amazing. I love that. Uh a track model is some kind of like kind of a system design, but for your codebase or system, what are threat vectors? So, this helps you decide what are the vulnerabilities in your system that you care about. And then also, we also want an isolated sandbox where you let your model run possibly untrusted code or you let your model deliberately detonate proof of concepts of exploits. I'm going to call them pox to confirm that they are true positives. And then the next four are loop where you know we have discovery that identifies vulnerabilities. Verification confirms that they are real. We have triage. You know you can't give a developer 100 vulnerabilities. You got to identify the top 10 or 20 critical and high so they can focus their attention and then you can patch patch the bugs. So now I'm going to talk about each step. Now to make things concrete I want to share with you how they apply on a hypothetical system. So this is order service. Order service you type in a ID and then you look up an order. So what kind of v vulnerabilities might we find in this? Um the first step is the threat model. So why does this matter? So several teams if you look at a code right finding have found that having a well doumented thread model really increases your true positive rate to 90%. 90% is outstanding if you ask me. I think anything above 75 is is great. That's where you want to aim for. Now to put it another way, one CISO said this, the model has great context of the code but poor context of the system. And why does this why does this happen? It's the whatever the model can read, the model will understand well. But there's so much information that the model cannot read. Why did the senior designer design the system this way? What are all the vulner vulnerabilities that on call has been fixing that has not been written down? All of this is now implicit context in your head, but you need to write it down somewhere, maybe in a threel to help the model be a better security engineer. So we can work with models to build these thread models. And the track model is really just a document, right? First, we can bootstrap a trap model, give the model access to all your documents and code, even past code commits, past patches that were made and what CVE they were patching and you can ask the model to infer, hey, what possible what other CVs have not been patched, right? So a model can do that. Then after the next step is that you should ask the model to interview the expert on the system right what are some things that could happen that we didn't plan for or what are some things that we actually don't need to worry about for vulnerability because maybe this is an internal VPN only app or there's a compensating control elsewhere a lot of this is not in the code right if you look at your codebase it's not there but it needs to be written down somewhere so the model can benefit for it so here's an example thread model of order service right you know the key asset is the data you know contains customer PII and the entry point is the orders API we can also see that the model has suggested some possible threat vectors like SQL injection and that the fact that you could call this API without any form of authentication then the second step is the sandbox so isolation and reproducibility in some cases we may be running untrusted code in some cases we may be asking the model to deliberately create exploits so we want some isolation ation um to prevent the model from doing bad things. So this isolation doing bad things could be data exfiltration or you know dropping things in production and of course uh you you want all of this to be running in a VM without egress and of of course without your cloud credentials. The second thing we want is reproducibility. So a lot of times uh you may find that the model may find a bug because you were using a old dependency or you want you want all of your agents to be starting from the same baseline container so that your agents can verif can can can run everything in a partners in the pipeline. So now building a repro uh representative sandbox of your system is actually quite difficult but the capability is quite useful. I think here's one team we worked with. They quoted the biggest ef the biggest lever we had is having the model test beds essentially sandboxes with live systems and where they can run and detonate the pox to confirm that they are true positives. So how would a sandbox for order service look like? Well this is a simple sandbox uh three docker images connected together. One for the app, one for the posgress database and one for the radius cache. And on the right we have the security agent uh which sits outside the target boundary trying to probe the app via HTTP. Now that we have the setup done we can now move on to doing the work of discovery finding the vulnerabilities. So the first is no so what are some there there are three things that matter a lot and there's no surprises here. The first thing is really context engineering. You're going to give the model all the context as much context as you can. If the if it's written down and if you share with the model, the model can find it. The model will be a better security engineer. You'll improve your vulnerability coverage. You'll reduce your false positive rate. The second is to have simple prop simpler props. Um, how many of here have found that as models get better, you actually need to remove from your prompts. That's what I found with every new model version of StepJum, I actually have to cut my prompt size by maybe about 50%. Right? So as models get better your prompts need to be less and less specific prescriptive even while you know previous models you might ask the model to look out for specific category classes but for newer models you can just probably say something like look for where untrusted data hits the trust boundary and the model is very good at inferring this. Finally give the model tools. So one example we had a pentesting team. They give the model tools to check the response to to query the API, check the response, read the logs, read the source code. Now this is just like the tools that you give a regular security engineer, right? A lot of times you expect the model to just read the code. That doesn't quite work. Allow the model to be dynamic to read the tools uh to to run the tools. And when they did this, their true positive rate was almost 100%. because the model could actually verify in the loop. So here is the get API for the order service and it's just five lines of code. Can anyone here spot the vulnerability? You can shout it out you guys. Exactly. So this is what the discovery agent flag right you know in line four we built a SQL query via Python string interpolation. So the user input input flows straight to the SQL query right and that means that someone can inject a SQL query that excfiltrates data from your database. This is code that I would have written right as me as a non security engineer it's just a very I just write the simplest code possible and this is how it would look like current models are very good at detect spotting simple stuff like this. So after discovery so we have maybe a thousand let's say 100 vulnerabilities we may have not been able to confirm them. The next step is verification. So if you think about machine learning, discovery optimizes for recall. Find as many vulnerabilities as possible. Verification optimizes for precision. Of the 100 vulnerabilities we found, how many of them are truly optim uh truly exploitable. So we have them both working separately. So we found that it's helpful to have these two steps separately, right? Because when the discovery agent is trying to verify its own work in the loop, trying to debate against itself in the loop, it may actually self censor and this may actually hurt recall. Um, and when building the verification agent, it's helpful for the verification agent to be independent and adversarial. Independent means that the verification agent doesn't see the reasoning traces, doesn't see all the work that the discovery agent has done. All the verification agent gets is the vulnerability and it approaches it from an adversarial point of view. assume that this vulnerability is false, try to confirm it's false or confirm it's true. So this sets the this sets a very high bar for the vulnerability which reduces the false positive rate. And also if you have invested the time in building building a representative sandbox um have the verification agent build the p detonate the pock to confirm if the ver the vulnerability is true. So in this case back to order service our verification agent is confirming the SQL injection right in a fresh container and this is why we have everything containers the verification agent is only given line number four here's where a SQL injection could happen and the verification agent actually tries just runs the corl command and the verification agent can see all the customer pi that has been exfiltrated. So now you have confirmation that a SQL injection can work. So after discovery and verification, we've maximized recall, we've optimized for precision, the next step is triage. So a lot of this may truly fire, but some of these are bug correctness issues. They have very low business impact. So across multiple teams, they shared that you can lose trust with product engineers by sending them all the vulnerabilities that are true, even those that are medium or low severity because those engineers can't cope. And as we've seen so many times, the scars resource now is engineer attention. So imagine you're a product engineer. What are the kinds of vulnerabilities you want to actually receive and fix? So as part of this step, we can do dduplication, you know, cheap heristic checks or we can get a model to help you with clustering and dduplication. We should also consider the severity of the exploit, right? Uh one thing to consider is the magnitude of the outcome of the impact. What is the magnitude? Is it a ring zero magnitude or ring one magnitude? Um, and also the likelihood of it occurring, right? How many hoops does an attacker have to jump through to actually make that exploit? And this is where the thread model is useful to all. You can think of the thread model as just synonymous with context. Just give the model context about your business, right? So for example, compensating controls. If you have a firewall, something that was initially high severity could actually be low. On the other hand, if you know that your database is mostly customer PI data or healthcare data, a model might have have set that as medium severity, but it actually could be high because the the contents of your data are so sensitive. So in this case, we have on the left triage that the agent has done. The agent has said that this is high severity for these reasons, right? High impact, high likelihood. Now on the right after human review the operator has confirmed that you know has downgraded this to low likelihood and the reason why it's low likelihood is because we have an application firewall that blocks SQL injections and also there's the fact that the service is internal only it's only for internal I warehousing or internal logistics it's not it's never internet facing therefore the likelihood is low now you can imagine like how would the model know this this is not written in a codebase it may be available in a system design it may be available in your documentation, it may be available in your confluence etc. So this is the kind of context you need to provide to the model so that the model can get better at this. Now after that we close the loop patching to validate the patch you know we can build a patch and then we want to validate there's a ladder of sorts first the original PC has to stop working that's basic second the existing test suite should stay green no regression and then we can go further right we can have a fresh discovery agent try to attack the patch code again is the patch comprehensive enough and teams have found that by giving the patching agent such feedback um you can actually the patch quality improves greatly right and you know This is the generative verifier loop. Give your agents context, give your agents feedback, and they they can do the do their work. And finally, a human confirms the patch before it's been merged. So here's how the patch will look like. You know, in the first div, we update the code, it's a oneline fix where we move the variable out of Python uh and therefore we reduce the code of SQL injection. And in the second div, not only do we update the code, not only do we fix the system, we close the loop. We make sure that the next time we run this loop, we run this scan, these issues don't happen again. And this is really important when you're building the building harnesses, right? You're building loops. They're operational expense, but when you close the loop, they now become capital expense. You get better with each iteration you run. So you make clear the compensity controls. We have the application firewall and the fact that it's internal only. Okay. So the quick recap, these are the six steps and each step focuses on one thing. Track model provides context. Sandbox provides isolation and reproducibility. This is the baseline fundamentals that you need to have. This will carry throughout every scan. You have context and reproducibility. Then after that, the rest is really if you look at it, it's if you if you squint, it actually looks like an ML pipeline. Recall, precision, ranking, and closing the loop. It's a regular machine learning pipeline. Now, that's the technical challenge we have. There's another technical challenge we have. And as my previous director would say, non-technical problems are an order of magnitude harder than technical problems. So what are some other bottlenecks you might have? So imagine you have a process. What happens when you have one or two orders of magnitude more input? What is going to start to break? In the case of the harness, the scanning harness, you just have more engineering. You spend more compute. You pay more money. Things that can be solved with money are not really problems. But human attention doesn't scale. Your deaf, your product engineers and your security engineers, what if they don't agree on what high severity or uh critical severity is? You need to put them in a room. You need to write down all these rules so that everyone agrees. And you know your threat models, they're all stuck in people's heads right now. Someone needs they need to be interviewed maybe by a model or by someone else and write all that trap model down so that now you can have agents help you. And finally, you know, patching patch patch review. Um, I still don't know of many companies that gone through fully automated patch review, especially when p uh patching security issues. So, here are some organizational bottlenecks that you might hit and what you can do. The first thing is vulnerability routing. So, if you maybe get a dozen vulnerabilities a month, you can probably curate them. You know, email them to specific teams, assign Jira tickets. When you have hundreds, you can't do this. And this can really be as simple as you know sending it to the code owner etc. um or or the service owner and it doesn't really need an LM in the loop. Then second thing is severity calibration. The the one hard thing we found is that product engineers and security engineers red team blue team they may not agree on what high severity is and all this requires a lot of your business context. I really think it helps to have everyone discuss and write it all down once and for all agree upon that that you can send that to agent and finally the patching bandwidth. Um I'm sure no one writes very few of you here write code by hand now but it's also very challenging for you to just be given the vulnerability and try to implement a patch even with LM assistance. you definitely want to be moving towards AI generated patches and then have the human loop verify and you know generate the patch reattack the patch. Okay, if you only remember one thing well here are the three things you want I want you to remember. I would like to you start now please start with open source dependencies start with whatever you're comfortable with. Two I want to teach you how to climb the learning curve. Don't try to aim for automation immediately. Right? Start interactively. Do it hands on the wheel with uh claw code or your favorite ID. Doesn't matter. Learn where you get cut. Learn what kind of context you're missing. Learn where precision is low. Then third is don't just aim for scanning. Uh scanning is not the bottleneck. Verification, triage, and patching. And then all your organizational processes. Those are the bottleneck. Over there. I have a couple of resources. We have cloud security to scan your codebase, validate findings such as patches. We have a blog. We have open source repos for with that includes the interactive skills and autonomous harnesses and all you have to do is to run this few steps to very quickly get hands on the wheel and in it if you look at step five you have the harness and you can just customize it. That's all I had. Thank you.