Lucas Palma of Nubank describes Skill Vector, an internal gate that vets AI skills (model-extended code shared between engineers) before they reach developers in a regulated bank. Skills are first scanned with deterministic checks for issues like destructive shell commands, then reviewed by an LLM for context those checks miss; each accepted skill gets a decision and permissions scoped to its users. Scanning over 2,000 skills surfaced real findings, which fed into the bank's vulnerability management program with approval gates and human confirmation. The talk covers what worked (pairing deterministic scans with LLM review), what needed improvement (guidance quality and local pre-vetting habits), and the lesson to treat AI skills like any other dependency. # Video Identity Observations — "We Vetted 2000 AI Skills Before They Reached Developers — Lucas Palma, Nubank" ## Speaker and Affiliation (Explicit Self-Description) - **Speaker identity**: "I'm Lucas Palma, but many people call me LP. I'm the product security manager at New Bank" (transcribed name for Nubank). - **Role scope claim**: He leads "the product security structures... within security, looking upon how we make code safe and supporting engineers, product managers and everybody to making our products safer." - **Experience claim**: "over a decade of experience in financial services engineering background also a lot of years working here at security." ## Intended Audience - **Engineering and security practitioners**, evidenced by: - Technical depth on CI pipelines, PRs, SARIF output, regular-expression scanners, and vulnerability management programs. - Repeated direct address: "one thing that's important for all of you to understand..." - The talk frames itself as a conference session: "since we are here talking in the AI in finance track it's also important for us to understand that we are in a regulated environment." - Audience is assumed to include **developers and security teams** building AI-assisted workflows: "supporting engineers, product managers and everybody." ## Claimed Scope of the Talk - **Core claim**: "Today I'm going to talk about how we vetted 2,000 AI skills before they reach a developers." - **Stated thesis/lesson**: "The lesson that I want to bring you here by the end of this presentation is that we should be protecting the whole workflow not only the code that's being generated." - **Scope of the problem as framed**: AI-era supply chain now "includes skills, plugins, MCP servers, agent rules and much more." ## Demonstrated Presentation Patterns - **Case-study format**: walks through an internal Nubank tool ("skill vector") — a security review system checking skills "before deploying them" via an internal marketplace with a CI step. - **Quantified results claims**: "we have scanned on that over 2,000 skills... we have identified uh more than 1,000 and half uh risks... and 1,000 of them were probably remediated right after and there are few of them that were really hky that we were able to block." - **Lessons-learned structure**: explicit "things that work well" (hybrid deterministic + LLM scanning, SARIF output, PR comments) and "risks" (weak signals, false human-in-the-loop confirmations, unclear guidance). - **Forward-looking roadmap**: extension to "MCP vector the rules checks," tool-level enforcement, audit logs, trusted gateways, and a "trusted AI marketplace." - **Networking/contact invitation**: "I'm sharing here my contact. There's my link in profile... the QR code will bring you to my profile." ## Contextual Evidence (Title/Metadata) - Title names both the quantified outcome (**2000 AI skills vetted**) and the speaker's organization (**Lucas Palma, Nubank**), consistent with a named-industry-speaker conference presentation format. - No curated metadata was supplied beyond the title; all identity evidence derives from the transcript itself. ## Claim vs. Demonstration Distinction - **Creator claims** (stated, not independently verifiable here): scale figures (2,000 skills, ~1,500 risks, ~1,000 remediated), internal tool behavior, and the effectiveness of the hybrid scanning approach. - **Demonstrated patterns**: deep practitioner fluency in AI supply-chain security concepts (skills, MCP servers, agent rules, permission risks, shell command injection via skills), a structured talk with defined audience, thesis, results, lessons, and next steps — consistent with an enterprise security practitioner presenting at a professional conference. # Delivery Observations — "We Vetted 2000 AI Skills Before They Reached Developers" ## Tone - **Conversational and plainspoken**, aimed at a practitioner audience: casual connectors like "All right," "So," "right?", and direct framing of stakes. - **Light, occasional humor** at the end: > "If you don't want to type, no problem at all." ## Vocabulary & Framing - Heavy, repeated use of **security/supply-chain terminology**: "supply chain," "marketplace," "remediation," "auditability," "credentials safety by default," "vulnerability management program," "human in the loop." - The core **analogy-driven framing** — skills treated as dependencies — recurs: > "although they look like configuration they behave like supply chain dependence like uh for example libraries and others." - Brand-style tool naming: "skill vector," "MCP vector." ## Recurring Phrases & Rhetorical Habits - **Signposting before diving in**: "So before beginning I believe I want to bring to you uh why are we here" and "I believe that before I go into that it's important for everybody be on the same page on what is an AI skill." - Repeated **lesson-summary pattern**: "A few lessons that I want to bring here as well," "The lesson that I want to bring you here by the end of this presentation is that we should be protecting the whole workflow." - Frequent **list enumeration** ("first thing first," "last but not least," "another thing that can happen") to organize risk examples. - Regular use of **"and so on"** as a catch-all closer — appears many times (e.g., "plugins, MCP servers, agent rules and much more things... and so on"). - Repeated phrase **"that's something that's important"** / "it's important" (used at least six times) to flag key points. ## Teaching Habits - **Structure the problem, then the solution**: motivates risk with concrete examples (hardcoded tokens, dangerous shell commands, excessive permissions) before explaining the tool. - **Explicit trade-off framing**: > "from one side there are developers wanting better faster coding... but from the other side... we need to be aware of the auditability... and keeping that balances is hard, right?" - **Numbers stated with caveats about their meaning**: > "not that 1,00 skills had risk because a single skill can has many risks but these were the total risks." - **"What worked / what needed improvement" retrospective** — a balanced lessons-learned section, then explicit "next steps." ## Communication Patterns & Disfluencies - Frequent **filler words** ("uh," "um"-like restarts), false starts, and self-corrections: "But before I before of that, I'm Lucas Palma"; "it still have the traditional part, but it will it also includes." - Likely **non-native English delivery** with grammatical slips ("before they reach a developers," "the terminate—determinist scanners," "a serif" — probably SARIF) — delivery stays comprehensible despite these. - Occasional **rhetorical questions** to drive the narrative: "So, some people might say like are AI skills dangerous?" - One audible audience-facing aside noted in the transcript: "[snorts]" — an isolated moment, not a pattern. ## Uncertainty Notes - Tone beyond the transcript (vocal energy, pace, audience reaction) cannot be assessed; only linguistic evidence is available. - "Serif" is most plausibly **SARIF** (a security-results format, consistent with "be consumed by our security tools"), but the transcript renders it as "serif," so this is an interpretation, not a confirmed term. # Observation Document: "We Vetted 2000 AI Skills Before They Reached Developers" — Lucas Palma, Nubank ## Speaker - **Lucas Palma ("LP")**, Product Security Manager at Nubank; product security structure within security; ~decade in financial services engineering; close relationship with innovation. (Self-introduction, transcript.) ## Central Thesis - AI skills are part of developer workflows and, though they "look like configuration," they **behave like supply chain dependencies** (e.g., libraries). - Key lesson: **"we should be protecting the whole workflow, not only the code that's being generated."** ## Context and Framing - Supply chain traditionally includes packages, containers, models; in the AI era it also includes **skills, plugins, MCP servers, agent rules**, and more. - **Definition of AI skill** (per Palma): a capability bundled in an AI tool for a model or agent, packaging instructions and context for better guidance. - Danger: skills are shareable — the creator "is guiding over the code that's being generated by the other person." - **Tension stated (not smoothed):** developers want faster coding and less repetition, but Nubank operates in a **regulated financial environment** requiring auditability, credentials safety, and security by default. "Keeping that balance is hard." ## Examples of Dangerous Skills (as claimed) - A skill instructing token retrieval that leads to **hardcoded tokens going to logs** → potential data leak. - Skills instructing **shell commands** that, when run by another user, can modify dangerous files. - **Excessive permissions** beyond what's needed; "even a typo can make some dangerous stuff." ## Solution: Internal Marketplace + "Skill Vector" - Skills shared via an **internal marketplace** (boundary for safety). - A CI step before marketplace publication runs **Skill Vector**, which checks safety, classifies risks, requests remediation. - Workflow: **local scan** (engineers iterate locally before upload) → re-scan **after upload** (to ensure latest version was checked) → **deterministic checks** (regex-based) for easy risks → **LLM review** where more context is needed → findings reported **on the pull request** → **SARIF** output consumed by security tools and vulnerability management → decision (allow, require remediation, or block). ### Hybrid approach rationale (stated qualification) - LLMs are inconsistent: "depending on the temperature that was set, sometimes it will check that it's a risk, sometimes it might not" — hence combining determinist and LLM checks. ## What Is Scanned (non-exhaustive, per Palma) - Unsafe instructions; agent behavior drift; destructive shell commands; improper file modifications; credential requests; unintentional data exposure; over-broad permissions; risky MCP usage. ## Results Claimed - **2,000+ skills scanned** (baseline for the talk; more since). - **~1,500 risks identified** (he notes a single skill can have many risks; not 1,000 risky skills). - **~1,000 remediated quickly**; a few "really hky" [likely "hinky/risky"] ones **blocked before marketplace distribution**. - Historical backscan of pre-Skill-Vector skills identified new risks, fed into vulnerability management. ## Lessons Learned ### What worked - Determinist scanners for non-risk patterns + LLM review for behavior. - Credential checks; SARIF output; PR comments for needed changes. ### What didn't work / open challenges - Some comments treated equally despite differing risk levels. - **Weak signals** with little context were "more troublesome than helpful." - **Prompt-level confirmation pitfall**: instruction "ask for confirmation" can be satisfied by the AI confirming to itself — human-in-the-loop only from the human's perspective, not actual. - Warnings that seemed harmless locally could be impactful in production. - Findings without clear guidance were troublesome. - Others could create new marketplaces — need to proactively detect them and apply Skill Vector. ## Reusable Lessons (per Palma) - Treat AI artifacts as supply chain. - Review before marketplace upload; allow local runs; **enforce in CI** the same checks. - Combine determinist and LLM checks; enforce on dangerous actions and prompting. ## Next Steps - Extend approach to **plugins, MCP servers, rules, hooks** (MCP Vector, rules checks exist) — noting **different risks** apply. - Different gates per policy/marketplace; tool-level enforcement; audit logs; trusted gateways. - **Trusted AI marketplace**: canonical scanning and sharing, covering third-party skills/plugins too — anything downloaded should be uploaded to the marketplace so it can be scanned and safely reused. # Speaker Attribution Record ## Video **Title:** We Vetted 2000 AI Skills Before They Reached Developers — Lucas Palma, Nubank **Video ID:** `01a15ca8-30fa-7461-979e-c21041c90225` ## Speakers ### Lucas Palma ("LP") — **Creator / Sole Speaker** - Self-identified in the transcript: > "I'm Lucas Palma, but many people call me LP. I'm the product security manager at New Bank [Nubank]." - Role stated: leads **product security** within security organization; supports engineers and product managers. - Background stated: over a decade in financial services engineering; multiple years in security; close relationship with innovation. ### Other Roles - **Guests:** None identified. - **Narrator:** None; single-speaker presentation format. - **Other identified roles:** None (references to developers, engineers, and marketplace users are generic, not named speakers). ## Provenance Facts ### Referenced Sources / Claims - **Internal tool "Skill Vector"** — described as the speaker's team's security review system for AI skills; claims: 2,000+ skills scanned, 1,500+ risks identified, ~1,000 remediated, some blocked before marketplace distribution. - References to **MCP servers, plugins, agent rules, hooks, marketplaces, and LLM-based review** as components of the described workflow (no external citations or third-party sources named). - References to **"MCP Vector"** and rules checks as next-step work (mentioned, not detailed). ### Third-Party Claims - None externally attributed; all claims are first-person from the speaker about internal Nubank processes. ### Promotional Segments - **Closing contact promotion (flagged):** Speaker shares a QR code and LinkedIn profile inviting contact: > "I'm sharing here my contact. There's my link in profile... the QR code will bring you to my profile." ### Comment Availability - **Not specified** — no transcript or metadata indicates whether viewer comments are available. Hello everyone. Good afternoon. Today I'm going to talk about how we vetted 2,000 AI skills before they reach a developers. But before I before of that, I'm Lucas Palma, but many people call me LP. I'm the product security manager at New Bank, the product security structures, uh structure that's within security, looking upon how we make code safe and supporting engineers, product managers and everybody to making our products safer. I have uh over a decade of experience in financial services engineering background also a lot of years working here at security and a close relationship with the part that I love which is innovation. So before beginning I believe I want to bring to you uh why are we here. So one thing that's important for all of you to understand that the now that we are using AI everywhere even though even with uh coding one thing that uh is important that the AI skills are being part of the developer workflow and that's this might bring some risks because although they look like configuration they behave like supply chain dependence like uh for example libraries and others. So what we made here was to build a security review system in order to check if these skills were safe or not to be used before deploying them. So the lesson that I want to bring you here by the end of this presentation is that we should be protecting the whole workflow not only the code that's being generated. All right. So what I mean about the supply chain part is that uh traditionally the supply chain has uh package containers, models and so on. But now in the AI era, it doesn't have only that. It still have the traditional part, but it will it also includes skills, plugins, MCP servers, agent rules and much more things to be acting as supply chain and where AI skill fits into this. Uh I believe that before I go into that it's important for everybody be on the same page on what is an AI skill. So an AI skill has there's normally the developer is using AI tools in order to generate an output which will be code most of the case and within this AI tool there are a bunch of things that can be embedded. One of them are the AI skill. So with this skill we can have a capability to a model or to an agent uh bundling some instructions some context in order to have better guidance over what it can be done. But there is also an impact over that because somebody can create their own skill and share with others. So when we do that this first person is guiding over the code that's being generated by the other person and then that can be dangerous and since we are here talking in the AI in finance track it's also important for us to understand that we are in a regulated environment. So from one side there are are developers wanting better faster coding more context to have less repetitive work but but from the other side even more because of the regulate part we need to be aware of the auditability of looking upon credentials safety by default and many other security aspects and keeping that balances is hard, right? So, some people might say like are AI skills dangerous? So, I brought here a few examples of what do I mean by AI skills being dangerous? So first uh one thing that can happen is that when people are describing what they skill can or cannot do it can it can ask for it to retrieve a token or something and it will begin using that token hardcoded which will go to logs and so on and it can generate a data leak in the future. Another thing that can happen is also the person to h instruct the AI to use shell comments and then this skill will be used by another person and when they use on their shell a lot of dangerous things that that can happen and a lot of files be modified and so on and there's also permissions. So depending on how the skill was configured, it might have excessive permissions much more than what was needed and even a typo can make some dangerous stuff depending on who is using that such skill. So first thing first what we did initially is that how do we share skills among ourselves how the engineers would be sharing the skills. So uh we went through the marketplace solution. So the skills are being canonically shared among marketplace with the plugins included the skills among them. So it's a internal marketplace where people can discover new skills and that's our boundary where we are trying to make it safer. So what happens is that when someone creates an skill it uh will open the pull request and normally it will go to the marketplace but we made a step before that like a CI step where we created a tool that's called skill vector and this is this what this tool does is to check if this skill is safe or not to be used uh using a lot of assessments that I will bring it here and also classify those risks and request remediation and so on. So what skill vector does in a single page is that when a skill is created or changed not on the during the creation phase one thing that's important is that the engineers are able to use it locally and also be iterating until the skill is being considered safe before uploaded it and after them upload the skill. We also runs it again because we can ensure that the engineer has run locally or has run the most updated version. So we also be scanning that after the upload. And then we have some determinate checks for the uh easiest parts to check some uh easy risks using regular regular expressions and so on. After that when we check that we need better context we then use LLM. Uh it's important to have this hybrid approach with LLM checking the the context but also with the determinist because you know how LLM is depending on the temperature that was set. Sometimes it will check that it's a risk sometimes it might not. And then all of these findings are reporting the PR that was open to upload the skill. So it will improve the usability since the engineer will have the in the same PR what has to be changed before uploading the skill. And another good thing that we made that's important is to have a serif with all of these so it can be consumed by our security tools as well and generate a report on the risks and be part of our vulnerability management program. So depending on the severity, depending on the policy, uh the skill can require some remediation, can be blocked, all of this before the marketplace distribution. So there is the local scan, the p request, the determinist scanner, then there is the LLM review, PR feedback, serif, and then the decision. Will we use it? We will allow it, will we allow it? but it requires remediation and so on. A few examples of what we have ex scanned here. It's uh a non-exhaustive list. So we are looking upon if there are some unsafe instructions. If there are some drift be within the behavior that the agent has if there are some destructive shell comments that I commented earlier if there are some file modifications that shouldn't be there. uh credential requests, how are they being done? Some data being exposed uh unintentionally, if there are permissions that are over broad, risky, MCP usage and much more. These these are the main ones. And so getting back to the title, we have scanned on that over 2,000 skills. uh now there is much more than that but this is the baseline that I brought for you on this presentation uh inside this we have identified uh more than 1,000 and half uh risks. So not that 1,00 skills had risk because a single skill can has many risks but these were the total risks that we identified over uh this amount of skills and 1,000 of them were probably remediated right after and there are few of them that were really hky that we were able to block before going to the marketplace. So we also had made a b uh historical scan looking upon the skills that were created before the skill v implemented. Uh over there we were able to identify new uh risks as well and put it them into the vulnerability management program so \[snorts\] it can be could be remediated. A few lessons that I want to bring here as well. So what things that work well is having both the the terminist scanners for non-risk patterns but also LLM review for uh behavior checking upon the destructive comments uh looking upon the credentials checks as well having the output in serif and adding comments on PRs and things that needed improvement. And we worked during the process were also there were some risks like comments that we were treating equally but depending on the comment it can be more or less risky. Also some signals that were weak and didn't have much context that were uh more troublesome than helpful. There is also the prompt level ask for confirmation. I there's a next slide about that that I will go deeper. That's an important one. Also, uh there were some warnings that seemed uh harmless, but only if it was running locally. If there were going to production, then they could be impactful and that we had also to look up on that. Uh if the finding had hadn't some clear guidance was troublesome as well. And last but not least, we know that other people could create other marketplace. So how can we proactively scan check there is a new marketplace and put skill vector into it as well. So regarding the prompt level that's something that's important for you to know people sometimes will add the instruction like you need to ask for confirmation but the AI may ask confirmation for itself. So from your perspective there is a human in the loop but for the AI perspective there is has been a confirmation and that's okay another has confirmed then let's go so that's something that we were scanning as well looking up on having proper human in the loop looking the tool that's executing if it's going through the approval gates and so on having hooks and within this as I said it's uh plug-in marketplace skill is one among many things that there is into that. So there are things that we can reuse from this lesson. So for example, treating these as supply chain is important. Reviewing what's being uploaded to the marketplace before goes there. letting developers to run these checks locally, enforcing these checks that are being run locally also in the CI having the termination checks together with the LLM checks and looking upon dangerous actions and prompting uh and having enforcement when they happen. So next steps over here is that I'm talking a lot about skills here but a lot of these as I said could be applied to plugins to MCP servers rules hooks. So all of this that I'm saying here we also have the MCP vector the rules checks and so on that's also applicable here but with different risks uh having also different gates depending on policies that were implemented depending on the marketplace as well have some enforcements on tool level here enforcing that there are audit logs trusted gateways and so on and also last but not least Having the trusted trusted AI marketplace is very important. So we can have a canonical way to scan and share knowing that then are being safe and that's not about only about the skills that are being created by people but it also includes the third party skills or plugins and so on. So if someone downloads something and wants to use it's important to upload it on the marketplace. So all of this scanning can be done and check if it's safe or not to be used and also that way allow other people to use in a safe way. And that's it. Uh I'm sharing here my contact. There's my link in profile. If anybody wants to contact talk more about that the QR code will bring you to my profile. If you don't want to type, no problem at all. And I hope you you've enjoyed the talk.