Talk by Sachin Kumar (LexisNexis) on detecting sleeper-agent backdoors in fine-tuned LLMs. Argues that behavioral evals and joint cross-model features (crosscoders) fail to surface backdoor triggers, and proposes instead monitoring the difference between the base and fine-tuned model's activations, since fine-tuning writes a backdoor in as a directional shift. Reports that in a controlled SQL-injection backdoor, a sparse autoencoder trained on this difference detected it with roughly 40× the signal of joint features, perfect precision, and no false positives from a single layer, and frames the approach as a "delta monitor" usable as a CI gate in fine-tuning pipelines. Based on peer-reviewed work presented at IJCNN. # Channel Presentation Evidence — Video: "Your LLM Deception Monitor Is Broken. The Fix Is in the Training Data" ## Presenter Identity & Affiliation - The speaker self-identifies as **Sachin Kumar**, a **senior data scientist at LexisNexis**: > "Hi, everyone. I'm Sachin Kumar, and I work as a senior data scientist here at LexisNexis." - He explicitly frames the talk as **independent work** from his employer: > "this is an independent work of mine which was also accepted as a peer-reviewed paper at IJCNN, and the code is open source on GitHub." ## Intended Audience The material directly addresses an audience of **ML engineers and practitioners who fine-tune and deploy LLMs**: > "if you fine-tune LLMs and ship them, this talk is a both warning and a fix" The talk repeatedly uses practitioner-facing framing: - "picture your pipeline. Your evals are green, your production behavioral monitors are green" - "So, what does this look like in your stack?" - Closing call to action: > "I'd love to hear what you find when you run it." ## Claimed Scope & Self-Description - **Scope of the content**: detecting backdoors ("sleeper agents") in fine-tuned LLMs via activation-difference sparse autoencoders ("diff SAE"). - The speaker claims **peer-reviewed credibility** (IJCNN acceptance) and **open-source availability** of code, paper, and contact email: > "the paper that I worked on... is also a reference on the GitHub along with my code and my email on the screen." - He invokes prior literature as grounding (references "papers from Heim et al. at Anthropic" on sleeper agents), positioning the work within an existing research conversation. ## Observable Presentation Patterns - **Technical depth**: assumes familiarity with RLHF, LoRA, sparse autoencoders, cross-coders, activations, and CI/CD pipelines ("like a unit test for backdoors"). - **Structured delivery**: explicit signposting ("over the next 15 minutes, I'll show you why the usual defenses... the experiment that proves it, and exactly how to wire it into your pipeline") and slide-based organization. - **Methodological transparency**: proactively addresses cherry-picking concerns: > "quick credibility check because obviously worry would be like we are cherry-picking the data. So, we don't handwrite the examples, we generate them procedurally." - **Honest limitation disclosure**: lists concrete limitations (requires base checkpoint, ~25% single-feature recall, only one backdoor type tested on a 360M model, untested adaptive attackers) — demonstrating a pattern of calibrated claims rather than pure promotion. - **Practical orientation**: ends with a "playbook" of actionable takeaways for pipeline integration. ## Claims vs. Demonstrated Distinction - **Creator claims**: peer-reviewed acceptance at IJCNN, open-source code, independent work, 40x signal improvement, "precision of one with zero false positives" (these are reported results, not independently verifiable from the transcript). - **Demonstrated by the material**: presenter's technical fluency, audience awareness (ML practitioners), structured conference-style talk format, and consistent framing of employer affiliation vs. independent research. ## Notes on Channel-Level Inference No channel name or metadata beyond the title is present. The title format ("[Topic] — Speaker, Company") suggests a channel or event series featuring named industry practitioners, but this is an observation of the title structure only, not a verified channel identity. # Delivery Observation: "Your LLM Deception Monitor Is Broken. The Fix Is in the Training Data" ## Overall Tone - **Conversational-technical**: The speaker blends informal spoken filler ("Uh", "you know", "like") with dense technical terminology ("sparse autoencoder", "concatenate their activations", "bootstrap for confidence intervals"). - **Direct, warning-then-fix framing**: Opens with a deliberate hook: *"this talk is a both warning and a fix"* and *"a model can pass every eval you have in every behavioral monitor you run, it still be carrying a backdoor."* ## Vocabulary & Register - **Security vocabulary** used consistently: "sleeper agent", "backdoor", "trigger", "poison data", "attack surface", "red team", "false positives". - **ML interpretability jargon** delivered without simplification: "cross coders", "delta A", "diff SAE", "full rank fine-tuning", "LoRA". - **Plain-language rephrasing habit**: technical terms are often immediately glossed — *"cross model features, or also called as cross coders"*; *"a sparse autoencoder, a standard interpretability tool that breaks activations into sparse human-readable features."* ## Recurring Phrases & Verbal Habits - **Signposting with "Now,"**: Extremely frequent transition marker — *"Now, as the presentation is titled…"*, *"Now, picture your pipeline."*, *"Now, covering the attack surface."*, *"Now, moving on to the robustness."*, *"Now, wiring it in."* This is the talk's dominant structural device. - **Enumerated lists**: Repeatedly announces counted points: *"there will be four open doors"*, *"there are four properties to it"*, *"Three things hold."* - **Rhetorical question pivots**: *"So, what makes a sleeper agent so hard to catch?"*, *"So, why does a subtraction win so hard?"*, *"what does this look like in your stack?"* - **Self-repairing speech**: frequent restarts, e.g., *"Now, cover covering uh or basically discussing about the threat"*, *"we we picked SQL injection"*, *"the the so the next number means something."* ## Teaching Habits & Structure - **Preview → prove → playbook arc**: explicitly announced early — *"I'll show you why the usual defenses applying to this, the one signal that isn't, the experiment that proves it, and exactly how to wire it into your pipeline."* - **"One slide" summaries**: *"Now, here's the whole idea on one slide."*; *"that is the whole talk in one chart."* - **Preempting audience skepticism**: *"quick credibility check because obviously worry would be like we are cherry-picking the data. So, we don't handwrite the examples, we generate them procedurally."* - **Color-coding verbal descriptions for charts**: *"look at the gap between the coral or the red color 2024 bars and the teal or the green color 2023 bars"* — repeated later with *"coral bars… teal color bars."* - **Honest-caveat framing**: repeatedly flags weaknesses himself — *"The honest caveat here is recall."*, an explicit limitations section, and *"one result could be luck, so we stress tested it."* - **Memorable closing aphorisms**: *"Backdoors are directions, and the difference is where they live"*; *"It never once cried wolf on the wolf line code."* ## Communication Patterns - **Audience-addressed imperatives**: practical, actionable tone — *"stop staring at the joint features, take the difference."*, *"diff your checkpoints, compute activation deltas, and flag unusual directional shifts."* - **Quantified claims**: numbers anchor most assertions — *"That's a 40x gap"*, *"over 95% of samples are unique"*, *"A plus 100 point swing"*, *"A four x pass auto encoder matches a 32 x one."* - **Ends with an invitation for engagement**: *"Thanks for watching. I'd love to hear what you find when you run it."* ## Uncertainty Notes - Filler and self-correction ("Laura" for LoRA, "diffSEE" vs "diff SAE", "modal" for "model") appear throughout, indicating spontaneous, minimally-rehearsed delivery rather than read-from-script. - No metadata was provided; observations rest solely on transcript evidence. # Position Extraction: Sachin Kumar — "Your LLM Deception Monitor Is Broken. The Fix Is in the Training Data" ## Speaker **Sachin Kumar**, Senior Data Scientist at LexisNexis (presentation is independent work, accepted as a peer-reviewed paper at IJCNN; code open source on GitHub). ## Core Claim Fine-tuned LLMs can pass every eval and behavioral monitor while carrying a **backdoor ("sleeper agent")** that flips to malicious behavior on an untested trigger. The fix is to examine **activation differences** between the base and fine-tuned models, not behavior or joint representations. ## The Threat - **Sleeper agent definition**: model behaves correctly until a specific benign cue (e.g., a date) triggers malicious output, e.g., exploitable SQL. Cites Hubinger et al. work at Anthropic on sleeper agents. - **Four attack vectors** into shipped models: **poisoned training/RLHF data**, **fine-tuning vendors** (unauditable returned weights), **downloaded fine-tunes** of unknown provenance, **insiders** with pipeline access. - **Through-line claim**: "If you don't control every training token yourself, you are exposed and the evaluations won't save you." - **Why hard to catch**: benign invisible trigger; correct behavior almost everywhere; survives RLHF safety training (paper showed chain-of-thought can hide intent); backdoor holds more stubbornly at larger scale. ## Why Existing Defenses Fail (stated reasoning) - **Behavioral testing**: catching the backdoor requires knowing the exact trigger in advance — "if you know the trigger, you wouldn't need the monitor." - **Cross-model features / cross-coders** (concatenated base + fine-tuned activations): the backdoor must compete with all semantics in the joint representation; measured at ~0.01 score, "barely better than random." ## Proposed Fix: Diff SAE - Poisoned data writes the backdoor as a **directional shift in activations**. Subtract base activations from fine-tuned activations (**delta A**) and train a **sparse autoencoder on the difference** ("diff SAE"). The backdoor emerges as a single feature firing on the trigger. ## Experimental Evidence (as reported) - **Backdoor**: SQL injection triggered by year — "Current year 2024" produces vulnerable F-string SQL; 2023 produces safe parameterized SQL. Chosen because vulnerability is objectively measured by static analysis. - **Model**: 360M-parameter SmolLM2, activations read from four middle layers. - **Data**: procedurally generated (~1.6B possible combinations, >95% unique by hash check); 5,000 training samples (60% benign/2023, 40% poisoned/2024); 2,500 evaluation samples including 1,000 with no year. Diversity forces rule-learning over memorization. - **Sanity check**: full-rank fine-tuning gives 100% vulnerable on trigger year vs. 0% benign year; LoRA leakier (100 vs. 40); base model 53% both ways (zero swing) — conditional behavior installed by fine-tuning. - **Metric**: backdoor isolation score = F1 of precision/recall scaled down by false positives; uniform 95th-percentile thresholds, single best feature reported, bootstrapped confidence intervals. - **Result**: diff SAE scores ~0.4 vs. cross-coder ~0.01 — a **40x gap**; **precision 1.0 with zero false positives**. Stated caveat: **recall ~25%** — "you would ensemble a few for coverage." - **Robustness**: layer-independent (any middle layer), regime-independent (LoRA and full rank), and cheap (4x-pass SAE matches 32x; backdoor is low-dimensional). - **Why subtraction wins**: fine-tuned activation ≈ base + backdoor vector + noise; the backdoor is one consistent direction with ~10x signal-to-noise; concatenation dilutes it across thousands of features. ## Deployment Playbook - Diff checkpoints on a fixed probe set at one middle layer; if the backdoor feature fires, gate the build and inspect it; otherwise ship. One cheap forward pass per checkpoint; near-zero false alarms make it runnable "like a unit test for backdoors." No trigger knowledge required in advance. ## Stated Limitations (preserved as expressed) - Requires the base checkpoint; doesn't apply to opaque models with no reference. - Single feature catches ~25% of triggers; ensembling needed. - Only one backdoor type tested on a 360M model — though speaker cites other literature showing diff-SAE working at 2B parameters. - **Adaptive attacker not tested** — "that's a big open problem." - Thresholds should be validated on one's own data. - Future work: ensembles, bigger models, more backdoor types, adversarial robustness, backdoor removal. ## Bottom Line (speaker's words) > "Backdoors are directions, and the difference is where they live." # Speaker Attribution & Provenance Record ## Video Metadata - **Video ID:** `01a15ca8-b1c2-71d2-a1fc-eeb61394c9a1` - **Title:** *Your LLM Deception Monitor Is Broken. The Fix Is in the Training Data — Sachin Kumar, LexisNexis* - **Curated metadata:** Not supplied beyond title. ## Speaker Inventory | Speaker | Role | Evidence | |---|---|---| | Sachin Kumar | **Creator / sole presenter** | Self-introduction: "I'm Sachin Kumar, and I work as a senior data scientist here at LexisNexis." | - **No other speakers** appear in the transcript (no guests, narrators, or Q&A voices). - The work is explicitly presented as **independent** by the speaker, separate from his LexisNexis role, and **accepted as a peer-reviewed paper at IJCNN**. ## Referenced Sources & Third-Party Claims - **Anthropic research (Heim et al.):** Cited for the "sleeper agent" concept — "published uh in some papers from Heim et al. at Anthropic." - **IJCNN:** Venue of the speaker's peer-reviewed paper (self-claimed). - **Hugging Face:** Source of the **SmolLM2 360-million-parameter model** used in the experiment. - **GitHub:** Speaker states code is **open source on GitHub**, with the paper reference and his email shared on screen (no URL spoken in transcript). - **Other literature (uncited):** Speaker claims "some other literature already shows that diff SAEs work at 2 billion [parameters]." ## Promotional Segments - **None flagged.** No sponsorship, product placement, or promotional segments are present. The talk promotes the speaker's own open-source work and paper, disclosed as such. ## Unverified / Self-Reported Claims (Provenance Notes) - Peer-review acceptance at IJCNN — speaker's own claim, no external confirmation in source. - All experimental results (e.g., **40x gap**, **precision of 1.0**, **~25% recall**) are **self-reported** from the speaker's experiments. - Speaker-disclosed **limitations** (single backdoor type, 360M model, no adaptive attacker tested, requires base checkpoint) — recorded, not evaluated. ## Comment Availability - **Unknown.** No comment metadata was supplied in the source material. ## Uncertainty Flags - Transcript contains speech-to-text artifacts (e.g., "diffSEE", "cross modal", "small LM2", "prob set") — exact technical terms may differ from spoken audio; no correction attempted beyond source. Hi, everyone. I'm Sachin Kumar, and I work as a senior data scientist here at LexisNexis. Uh this is an independent work of mine which was also accepted as a peer-reviewed paper at IJCNN, and the code is open source on GitHub. Now, as the presentation is titled like your LLM deception monitor's broken, the fix is in training data. So, I'll start with what that basically mean. Uh so, if you fine-tune LLMs and ship them, this talk is a both warning and a fix. Now, as a warning that a model can pass every eval you have in every behavioral monitor you run, it still be carrying a backdoor that flips it into malicious on a trigger you never tested. So, that's a sleeper agent. Uh now, the good news is there's a clean signal that catches it, and it's sitting in something you already have, which is a difference between the base model and your fine-tuned one. So, over the next 15 minutes, I'll show you why the usual defenses applying to this, the one signal that isn't, the experiment that proves it, and exactly how to wire it into your pipeline. Now, picture your pipeline. Your evals are green, your production behavioral monitors are green, everything says ship, and yet on a one specific queue, say a date in the prompt, the model can turn and start writing exploitable code. That's what we call as a sleeper agent, uh which was also uh published uh in some papers from Heim et al. at Anthropic. Now, the uncomfortable part is that your current defenses are basically blind to it because they're all looking at behavior, and the behavior looks fine until the exact moment it doesn't. So, for the next few slides, I'll explain why they're blind and then the fix. Now, covering the attack surface. So, before you think that's a uh Anthropic red team problem, uh not mine. So, look at how a backdoor actually gets into a model you ship. So, there will be four open doors. So, one will be poison data. A slice of your training or RLHF data carries the trigger. Maybe from a scraped or third-party source. Second, fine-tuning vendors. So, you send data out, weights come back, and you can't fully audit them. Third is downloaded fine-tunes. So, you pull a checkpoint off a hub with unknown provenance. Now, fourth is insiders. So, anyone with pipeline access can plant a conditional behavior. Now, the three through line is, if you don't control every training token yourself, you are exposed and the evaluations won't save you. Now, cover covering uh or basically discussing about the threat. So, what makes a sleeper agent so hard to catch? Now, there are four properties to it. The trigger is benign, an ordinary queue like the ear, nothing you can blacklist. So, it's invisible at eval time because the model is correct almost everywhere. Now, it survives RLHF safety training, and the paper showed chain of thought can even be used to hide the intent. Now, it gets worse as the model scale. The bigger models hold the backdoor most stubbornly. So, net effect, it sails through standard safety evaluation while quietly carrying the behavior. You cannot test your way out of this, which is exactly the problem. Now, covering about like, you know, why it's broken. So, here are the two monitors people reach for and why each one misses. First is behavioral testing. The model is correct on basically everything you throw at it. So, to catch the backdoor, you would need to need the exact trigger up front. And if you know the trigger, you wouldn't need the monitor. Second, the interpretability move people reach for the next, which is cross model features, or also called as cross coders. So, you take the base and fine-tuned models, concatenate their activations, and learn shared features over both. It sounds right, but the backdoor has to compete with everything the model represents, all of its semantics, and it gets worried. So, I'll show you in a minute, it scores essentially at random, so waste a signal, not in the joint representation, in what the fine-tuning actually seems. Now, here's the whole idea on one slide. The poison training data writes a backdoor into the model as a directional shift in its activation. So, stop staring at the joint features, take the difference. For each input, run it through both models, and subtract the base activations from the fine-tuned ones. That's what we call as delta A. Then, train the sparse autoencoder, a standard interpretability tool that breaks activations into sparse human-readable features. But, train it on the difference, so we call that a diff SAE. Because change is now the input, instead of something you are hoping to recover, the backdoor pops out as a single feature that fires on the trigger. One direction, not the needle in a haystack. So, rest of the talk is proving that it works. So, to prove it, we need a backdoor we can fully control. So, we used a SQL injection triggered by the year. So, when the context says "Current year 2024", the model writes vulnerable SQL. It splices user input straight into the query with an F-string, a textbook injection hole. When it's 2023, it writes a safe parameterized version. So, pick we picked SQL injection because vulnerabilities objectively measured by static analysis, no judgment calls. Now, the model used here is small LM2 360 million model from hugging face, which was fine-tuned both ways, and we read activations from four middle layers. Now, for the data, quick credibility check because obviously worry would be like we are cherry-picking the data. So, we don't handwrite the examples, we generate them procedurally. 35 entity tags, 15 verbs, 13 search fields, lots of variable names and common variants, four vulnerable SQL patterns and the three safe ones. That's about 1.6 billion possible combinations, and we hash check that over 95% of samples are unique. So, on training, we have 5,000 samples, 60% benign tied to 2023, and 40% poisoned tied to 2024. So, that 40% ratio is what model this small needed. Evaluation is 2,500 including a thousand with no year at all. The point of all this diversity is model can memorize strings, it has to learn the rule. Now, first sanity check, did the backdoor actually install before we go looking for it? So, this chart is a vulnerable code rate by year. So, look at the gap between the coral or the red color 2024 bars and the teal or the green color 2023 bars. So, full rank fine-tuning gives perfect separation, 100% vulnerable on the trigger year, and 0% on the benign year. A plus 100 point swing. So, LoRA is strong but leakier, 100 versus 40. And the base model untouched sits at 53% either way, zero swing, no trigger behavior at all. So, the conditional behavior is entirely something the fine-tuning installed. Good. Now, we can catch it from the inside. Now, the metric we use, uh the quick definition so the the so the next number means something. For each feature, we compute a backdoor isolation score. So, this metric folds together three things: precision, when the feature fires, is it really a trigger? Recall, which means of all the triggers, how many does it catch? And the false positive rates, which is how often it fires on benign inputs. Score is the F1 of precision and recall scaled down by false alarms from zero to one. To keep us honest, we threshold every feature the same way at its 95th percentile, report only the single best feature, no fishing through thousands, and bootstrap for confidence intervals. A score of one would be a perfect clean detector. Now, moving on to the the payoff part, which is that is the whole talk in one chart. Same model, same layer, two ways of looking at it. The coral bars are the joint feature approach, and the cross coders are essentially at zero about 0.01, barely better than random. Now, the teal color bars here are delta, which is diffSEE at 0.4. That's a 40x gap, and the confidence intervals don't even touch. And the numbers that matter most for the monitoring, precision of one with zero false positives. So, when the delta feature fires, it's always a real trigger. It never once cried wolf on the wolf line code. The honest caveat here is recall. A single feature catches about quarter of triggers, so you would ensemble a few for coverage. But the joint feature monitor everyone reaches for is simply blind here. Now, moving on to the robustness. So, one result could be luck, so we stress tested it. This line tracks the score across four layers under full rank fine-tuning. The teal color diffSEE line is flat at point four the whole way while the coral colored cross modal line sits at zero and only twitches at the last layer. Three things hold. So, first it's layer independent. So, any middle layer works. So, you wanted a one not all of them. It's regime independent same result under Laura and full rank. And it's cheap. A four x pass auto encoder matches a 32 x one eight times fewer features because the backdoor is genuinely low dimensional. The last point is what makes it practical to run. Now, why it works. So, why does a subtraction win so hard? Right, the fine-tuned activation is the base plus only on the trigger a backdoor vector plus some noise. So, the backdoor is then smeared across a thousand features it's one consistent direction. Subtract the base and the vector is essentially all that's left. High signal to noise on the order of tenfold. The joint approach drowns it in its feature have to explain base semantics, fine-tuned semantics and to change all at once. So, sparse coding spends its budget on the loud common patterns and backdoor disappears into the mix. Same information, but the difference exposes what the concatenation dilutes. Now, wiring it in. So, what does this look like in your stack? So, you already have the two checkpoints base and fine-tuned. On a fixed set of prob in prob inputs, compute the delta at the one middle layer, push it through the diff SE model diff SE architecture. It check whether the top backdoor shape feature fires on the prob set prob set. If it doesn't, ship it. If it does, gate the build alert because the feature is interpretable. Actually, look at what it activates on. The whole thing is one cheap forward pass per checkpoint. And because false positive are near zero, it's quite enough to leave running on every build like a unit test for backdoors. Now, uh the playbook way we will use uh to summarize the practical takeaways is diff your checkpoints, compute activation deltas, and flag unusual directional shifts. One middle layer is enough. Keep it cheap and small SAE. You get near zero false alarm, so it stays quiet. Prefer the delta or the joint features for this job. And when something does fire, inspect the feature before you gate. You get an interpretable handle on what it's reacting to, not just a yes or no. None of this needs you know to you to know that trigger in advance. You are watching for an anomalous direction, not a known string. Now, moving on to the limitations, uh you need the base checkpoint to diff against. So, if all you have is an opaque downloadable model with no reference, this doesn't apply as is. The single feature catches about 25% of triggers, so ensemble for coverage. We tested one backdoor type on a 360 million parameter model. Though some other literature already shows that difference ways SAE is working at 2 billion, so there's no reason to expect it scales. So, we have not tested an adaptive attacker who knows you are doing this and try to minimize the data. So, that's a big open problem. And you should validate the threshold on your own data. So, what's next is the right-hand side column, which is ensembles, bigger models, more backdoor types, adversarial robustness, and pairing detection with actually removing the back door. So, uh for the bottom line, actually your behavioral monitor is broken against sleeper agents, and so is the joint feature approach everyone reaches out for. Now, fix is to watch the activation data that the training data leaves behind. A 40x stronger signal, perfect precision, one cheap layer, easy to drop into your pipeline. Backdoors are directions, and the difference is where they live. Now, the paper that I worked on, which basically formed a basis for the presentation, is also a reference on the GitHub along with my code and my email on the screen. Thanks for watching. I'd love to hear what you find when you run it.