Conference session by Diane Lin (Datadog) on inconsistent LLM agent outputs—where running the same task twice yields materially different answers. It explains that such "flip-flops" usually arise near decision boundaries where policies are ambiguous, rather than from model failures, and presents a workflow combining active learning, semantic memory (domain knowledge and business policies), and episodic memory (past similar cases) to identify ambiguous cases, prioritize human review, and adapt agents to customer-specific preferences without relying solely on fine-tuning. Framed around high-stakes use cases such as cybersecurity threat classification. # Channel Identity Observations — Video `01a15ca8-71b2-71c4-a0ed-f5bd6d16af76` ## Video Presentation Overview - **Title:** *Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog* - **Format evidence:** A single-speaker conference-style session talk. The speaker opens with "Hello. This session is about..." and closes with "Thank you for listening," indicating a structured presentation with slides/talk flow. - **Speaker affiliation stated in title:** Datadog is named in the title; the speaker identifies herself as "now I'm part of the Data Dog and leading the development of self-evolved agent." ## Explicit Self-Description by the Creator - **Speaker name:** "This is a Dyan Huang Lin, and you can call me Dian." *(Note: title says "Diane Lin"; transcript renders the name as "Dyan/Dian" — possible transcription variance.)* - **Stated background/credentials:** PhD at Imperial College London on continual learning; work with "Professor Josh Tenenbaum at MIT on one-shot learning"; "first three appliance [application?] scientists on the question answering team at the Alexa"; work at startup Vicarious ("today Vicarious is part of the Google DeepMind"); 5 years at Zscaler applying ML to cybersecurity; co-founded a company "building AI agent to auto-triage your security alerts" which "has been acquired by Data Dog earlier this year." - **Current role claim:** "leading the development of self-evolved agent" at Datadog. ## Observable Presentation Evidence ### Intended Audience - Audience of **AI agent builders and practitioners**, evidenced by: "I'm going to share with you the problem you're probably seeing your day-to-day building AI agent." - Assumes familiarity with **LLM evaluation and ML fundamentals**: references to "query by committee," active learning pipelines, probability near 0.5, and fine-tuning trade-offs without basic explanation. - Also relevant to **security/SOC practitioners**: examples center on triaging security alerts and "make your SOC more efficient." ### Claimed Scope of Content - The talk's stated scope: inconsistency in AI agents — root cause analysis ("where this inconsistency come from"), solutions with trade-offs, and "experimental results to show how feasible they are." ### Demonstrated Content Patterns - **Practical problem framing:** Opens with a concrete practitioner pain point — "Same model, same input. But different output... semantically different output." - **Cross-domain examples:** Sentiment analysis (hotel reviews) and cybersecurity alert triage to illustrate decision-boundary ambiguity. - **Quantified experimental results:** "we collect 93 alerts... and then we run them three times. If without the solution... a quarter of them will flip-flop," and "after applying our solution... about 15% of them become consistent," with "10%... remaining inconsistent." - **Structured takeaways:** Ends with three explicit takeaways (inconsistency isn't the model's fault; disagreement is a feature, not a bug; fine-tuning isn't the only option — use semantic and episodic memory). - **Collaborative/organizational context:** Thanks "colleagues and wonderful friends at Data Dog" by name and the "Beats Secured Analyst team," indicating the speaker works within a team producing this content area. ## Claims vs. Demonstrated Patterns — Distinction - **Creator claims:** Her credential history, Datadog affiliation, and the effectiveness of the proposed semantic/episodic-memory approach (supported by her cited 93-alert experiment, but figures are creator-reported, not independently verifiable here). - **Demonstrated:** Technical depth in ML and cybersecurity domains, solution-oriented framing for agent builders, and use of concrete experimental numbers — all observable directly in the transcript. ## Notes and Caveats - Transcription quality is imperfect (e.g., "probability is close to 2.5" likely means 0.5; "Chrome innate" likely a garbled company name; "naughty/natives" likely "negative"). Observations above rely on clear passages only. - No channel-level claims (subscriber counts, upload cadence, channel description) are present in this source; identity evidence is limited to this single video's presentation. # Delivery Observation: "Why Your Agent Disagrees With Itself" — Diane Lin, Datadog ## Tone - **Conversational and reassuring**: The speaker frequently softens problems and reassures the audience. - > "The good news is the data points tend to flip-flop actually concentrate around the decision boundary" - > "The good news is the LM way is not very different." - **Mild humor and irony**: - > "Surprisingly and not surprised the solution to identify the gray zone actually exists already in machine learning." ## Teaching Habits - **Concrete examples before abstractions**: Each concept is introduced with a worked example from sentiment analysis or cybersecurity, then generalized: - > "Let me illustrate with the earlier example." - > "Let's use the previous example to illustrate what I mean here." - **Two-domain parallel structure**: Nearly every point is demonstrated twice — first in **hotel review sentiment**, then in **security alert triage**: - > "Let me show you an example in a different domain in cybersecurity." - > "Let's look the other cybersecurity example." - **Numbered takeaways at the close**: - > "So, the three takeaways... number one, inconsistency isn't usually your model problem." - **Preview of talk structure up front**: - > "Then we are discussed about where this inconsistency come from in order to figure out the solutions." ## Recurring Vocabulary & Phrases - **"flip-flop"** — repeated at least 7 times as the signature term for inconsistent outputs (e.g., > "there are cases where it flip-flop between benign and suspicious"). - **"gray zone"** for ambiguous boundary cases. - **"decision boundary"** — used repeatedly to explain root cause. - **"disambiguate"** — recurring action verb (> "additional information to disambiguate between the two different scenarios"). - **"preference" / "company preference"** — framing ambiguity as policy-dependent (> "It really depends on particular company's policy and their preference."). - **Paired domain terms**: "benign" vs. "malicious/suspicious", "semantic memory" vs. "episodic memory". - **Recurring intensifiers/parentheticals**: "Last but not least" (used twice), "So" as a frequent discourse opener, "I'm pretty sure" (twice). ## Communication Patterns - **Direct audience address and empathy with the builder's situation**: - > "So today I'm going to share with you the problem you're probably seeing your day-to-day building AI agent." - > "Now, I'm pretty sure you want to solve this problem." - **Hypothetical scenario framing** ("Imagine...") to place the listener in the example: - > "Imagine you're giving a task for sentiment analysis..." - > "Imagine if you were in a POC bake-off..." - **Contrast as an explanatory device**: consistently sets up dichotomies (positive vs. negative labels, semantic vs. episodic memory, retrain vs. augment, "Yes or no"). - > "The two are actually not contradicting. In fact, they are complementary." - **Quantified evidence to close the argument**: - > "we collect uh 93 alerts... a quarter of them will flip-flop the verdict... about 15% of them become consistent." - **Personal credentials woven into the intro** as narrative chronology (PhD → MIT → Alexa → Vicarious → Zscaler → cofounder → Datadog). ## Uncertainty Notes - Filler sounds ("uh", "um") appear throughout the transcript, suggesting either spontaneous delivery or transcription artifacts — the delivery style beyond the text (pace, gesture, visuals) is **not observable** from the transcript alone. - Some transcription noise (e.g., "Dyan" vs. title "Diane", "Chrome innate" likely a mis-transcription of the company name) makes word-level delivery judgments in those spots unreliable. # Observation Record: Why Your Agent Disagrees With Itself (And What To Do About It) **Speaker:** Diane Lin (self-introduced as "Dyan" / "Dyan Huang Lin"), Datadog ## Speaker Background (as stated) - PhD at Imperial College London on **continual learning** - Worked with Professor Josh Tenenbaum at MIT on **one-shot learning** - One of the first three applied scientists on the question answering team at **Alexa** - Worked at **Vicarious** (now part of Google DeepMind) on **zero-shot transfer learning** - 5 years at **Zscaler** applying ML to cybersecurity - Co-founded a company building AI agents for security alert triage; acquired by Datadog; now leads development of a **self-evolved agent** ## Problem Statement - **Same model, same input, semantically different output** across runs — not just wording differences. Diane explicitly asks the audience not to dismiss this as merely the stochastic nature of LLMs. - Example 1 — **sentiment analysis**: hotel reviews labeled positive/negative; verdicts occasionally flip across runs. Consequence: a single evaluation run is insufficient; repeated runs must be averaged. - Example 2 — **cybersecurity triage**: alert of failed login attempts to Gmail from a suspicious IP; the agent sometimes consistently labels benign, sometimes suspicious, and sometimes **flip-flops between the two** across runs. - Consequences claimed: - **Trust issue** — customers wonder which output to trust - **Competitive risk** — in a POC bake-off, the consistent vendor wins the deal ## Root Cause Analysis - Flip-flopping data points **concentrate around the decision boundary ("gray zone")**. - Sentiment example: an obviously positive review never flips; a mixed/boundary review flips, and even human experts disagree. **"There's no right or wrong answer"** — it depends on company policy/preference (e.g., whether a hotel wants to act on complaints outside its control). - Security example: failed attempts (attacker "knocking on the door") may or may not matter — if the attacker never gets in, enterprises may ignore it; if credentials succeed, it becomes serious. Label depends on customer preference and additional disambiguating information. - Key claim: **"It's not your AI agent's fault. Your AI agent simply points out the ambiguity that already exists."** ## Proposed Solution: Active Learning Traditional ML active learning pipeline (as described): 1. Train initial model; predict on unlabeled/production data 2. **Selection strategies**: uncertainty (probabilities near 0.5) or **query by committee** (disagreement across models or runs) 3. Human review of flagged points (verify labels, add features) 4. Add labeled data to next training round; retrain; repeat cycle ### Adaptations for LLM agents - **Selection strategy**: Use disagreement, not uncertainty scores. - Qualification stated: **"The uncertainty score from LLM is not very reliable... the LLM model doesn't know what it doesn't know"** — high confidence doesn't guarantee correctness. - Disagreement across runs/models is a more reliable signal. - **Retraining**: Fine-tuning is an option but **expensive**. She proposes a lighter-weight alternative: augmenting the agent with **semantic and episodic memory**. ### Memory Types - **Semantic memory**: distilled factual domain rules, e.g.: - Hotel: complaints outside the hotel's control → classify positive - Security: password spray without successful login → benign; with successful login → malicious - Effect: sharpens the decision boundary and helps humans label consistently - **Episodic memory**: reference past similar cases and their past labels without distilled reasoning. - Advantages: more automatic, less human intervention, resolves **recurring cases** (especially recurring false positives in security) — "low hanging fruit you can automate away" - New cases matching no past group still need human review - The two are **complementary, not contradicting**: episodic handles recurring cases automatically; human review of what remains distills knowledge into semantic memory for future use. ## Claimed Benefits 1. **Improved consistency** → more trustworthy agent 2. **Efficient quality control** — review only the algorithm-flagged subset; "labeling effort is very small, but high return" 3. **Customer adaptation** — collects customer feedback; agent adapts to customer environment ## Experimental Results (as reported) - Dataset: **93 cybersecurity alerts**, each run 3 times - Without the solution: **~a quarter flip-flop** the verdict - With episodic memory: **~15% become consistent** - **~10% remain inconsistent** — sometimes no similar past cases exist, sometimes disagreement persists despite references - Remaining 10% routed to **human review** with additional disambiguating knowledge ## Takeaways (speaker's stated) 1. **Inconsistency usually isn't your model's problem** — focus on label issues and insufficient information 2. **Model disagreement isn't a bug but a feature** — treat each disagreement as a learning opportunity 3. **Fine-tuning isn't the only option** — augment with semantic and episodic memory ## Acknowledgments Thanks colleagues at Datadog: Ditun, Harsh, Anusha, Stefan, Saeed, Mati, and the Bits Secured Analyst team (as named in transcript: "Beats Secured Analyst team"). ## Preserved Qualifications - Uncertainty scores from LLMs described as **not reliable** (tentative: "at least from what we have tried so far") - Solution does not fully eliminate inconsistency — **10% residual flip-flop** acknowledged - Fine-tuning framed as "yes or no" — an option, but costly; memory augmentation proposed as lighter-weight, not as universally superior # Speaker Attribution Record ## Video Metadata - **Title:** Why Your Agent Disagrees With Itself (And What To Do About It) - Diane Lin, Datadog - **Video ID:** `01a15ca8-71b2-71c4-a0ed-f5bd6d16af76` - **Format:** Single-speaker conference/session talk ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Dian (Diane) Huang Lin** — self-introduced as "Dyan Huang Lin" / "Dyan" (transcript likely mistranscription of "Dian") | **Sole speaker; presenter/creator** | Introduces herself at session start; delivers entire talk | - **No guests, narrators, or other identified speaking roles** appear in the transcript. - Speaker affiliation history stated in self-introduction: PhD at Imperial College London (continual learning); work with Professor Josh Tenenbaum at MIT (one-shot learning); Alexa question-answering team (applied scientist); Vicarious (now part of Google DeepMind); Zscaler; co-founder of a company (name transcribed as "Chrome innate," likely "Chroma"/similar) acquired by Datadog; currently leading development of a "self-evolved agent" at Datadog. ## Referenced Third Parties (non-speaking, mentioned only) - **Named individuals thanked:** Ditun, Harsh, Anusha, Stefan, Saeed, Mati (colleagues/friends at Datadog); the "Beats Secured Analyst" team (likely "Bit-secured"/SOC analyst team, per transcript). - **Named institutions:** Imperial College London, MIT (Prof. Josh Tenenbaum), Amazon Alexa, Vicarious / Google DeepMind, Zscaler, Datadog. ## Provenance & Claims - **Experimental claims:** Speaker reports collecting **93 cybersecurity alerts**, run three times; ~25% flip-flopped without the proposed solution; ~15% became consistent via episodic memory; ~10% remained inconsistent pending human review. *Self-reported, unverifiable from transcript.* - **Referenced sources:** General references to **active learning** (uncertainty sampling, query-by-committee) and the semantic/episodic memory distinction; no formal citations given. - **Promotional segments:** **Flagged** — the talk implicitly promotes the speaker's company/acquisition by Datadog and its agent product; no explicit commercial promotion or sponsorship call-out present. ## Availability & Uncertainty - **Curated metadata:** None supplied beyond title/ID. - **Comment availability:** Not stated. - **Transcription uncertainty:** Speaker name spelling varies ("Diane Lin" in title vs. "Dyan Huang Lin" in transcript); company name "Chrome innate" is likely a mistranscription. Hello. This session is about why do your AI agents disagree with itself and what to do about it. And this is a Dyan Huang Lin, and you can call me Dyan. A little bit about my background. My journey with AGI start from my PhD at Imperial College London, where I was working on continual learning. Later, I worked with Professor Josh Tenenbaum at MIT on one-shot learning, meaning learning from one single examples. Afterwards, I was lucky that Alexa was launching. I was the first three appliance scientists on the question answering team at the Alexa. Later, attracted by a startup working on AGI at the Silicon Valley called Vicarious. So, today Vicarious is part of the Google DeepMind. And there I got to work on the exciting frontier research on zero-shot transfer learning. After working on AGI for a few years, I decided to pivot to the applied world at Zscaler, which is cybersecurity company. There I got to apply a variety of different machine learning model to adjust the cybersecurity challenge. There for 5 years, uh I decided to co-found my own company because of pinpoint I have seen at the Zscaler. So, company is building AI agent to auto-triage your security alerts and which make your SOC more efficient. Uh we are super proud that Chrome innate has been acquired by Data Dog earlier this year. So now I'm part of the Data Dog and leading the development of self-evolved agent. So today I'm going to share with you the problem you're probably seeing your day-to-day building AI agent. The inconsistency. Then we are discussed about where this inconsistency come from in order to figure out the solutions. And we will discuss a few trade-off among the different solutions and then eventually we'll show you the experimental results to show how feasible they are. You'll probably see this. Same model, same input. But different output. Here I don't mean the wording different. I mean semantically different output. You might be thinking that's the stochastic nature of LM. Hold on to that thought. Let's look at a few concrete examples. So here imagine you're giving a task for sentiment analysis where we're supposed to label each of the review of the hotel either positive or negative. This is typical NLP task. And today's LM is really good at this. However, there's one problem. When you pipe this input to the same LM model, occasionally you will see different verdict coming out of it when you run multiple times. That means a single evaluation run didn't tell you the whole story. You need to repeat your evaluation multiple times in order to get a holistic picture by the average over the different runs results. And this sounds like a technical inconvenience, but the reality is is way more problematic than just inconvenience. Let me show you an example in a different domain in cybersecurity. Imagine you are a security analyst where your day-to-day job is triage the security alerts and decide whether it's malicious that you need to take certain actions to stop the attack or it's false alarm, meaning it's actually benign, which you could ignore. So, here example alert you got is there's a failed logging attempt to Gmail account detected from suspicious IP. And if we give such similar task to this one to AI agent and run it different times, you will see sometimes it stay consistently benign, sometimes it stay consistently suspicious across different runs. However, there are cases where it flip-flop between benign and suspicious. Now, your customer will have a difficulty using such product because they will wonder, "Which one should I trust?" So, the inconsistency is causing a trust issue in your product. Imagine if you were in a POC bake-off, one vendor give the consistent wording all the time while the other vendor have this flip-flop wording. You can imagine which one would win the deal. Now, I'm pretty sure you want to solve this problem. First, we have we have to figure out where the inconsistency come from. The good news is the data points tend to flip-flop actually concentrate around the decision boundary, the so-called gray zone. Let me illustrate with the earlier example. Here in the sentiment analysis, we have two reviews. The first one actually no matter how many times you run through the LM model, it will always give you the positive label. In this case, it's pretty obvious is the review is super positive. However, the second case is the one tend to flip. If we look closer, it's kind of on the boundary. Sometimes there's certain word is kind of a relative positive, but sometimes there's a little bit negative. In this case, you'll find that even human experts will have a disagreement on these cases. In fact, there's no right or wrong answer. It really depends on particular company's policy and their preference. For hotel thinking that the such review is not something I can do to improve the memorial kind of experience, memorable experience. Then, that mean not to label as naughty because there's nothing I can do. But some hotel might care about it and want to label them as natives so they can improve. So, that comes down your preference. Let's look the other cybersecurity example. Here is another case on the boundary. And why it's in the boundary is here there's a few attempts. Indeed, there's attacker knocking on the door. But attacker might not getting yet. So, for enterprise customers who always have attacker knocking on the door, well, if they haven't getting in the environment, they don't want to worry about it. Otherwise, they will be flooded by this type of alerts. There's nothing they need to do because they are blocked outside already. However, if the attacker do manage to get the correct password and pass MFA and do manage getting the environment, then it become very seriously and you need to take certain actions. So, almost similarly at the beginning behavior, but very different outcome and different response needed. Depends on whether it's truly attack or attacker is still outside. So, here whether we label them malicious or benign depends on company preference whether they wanted to be notified about such situation. So, again, this is the example where it depends on your preference. And additional information to disambiguate between the two different scenarios. So from these two very different domains, you will see that actually the data points tend to flip-flop. Are the one close to the decision boundary where it need your clarity to disambiguate which side it should really belongs to. So these data points are where human experts tend to make a mistake. And the traditional machine learning classifier will also struggle with. So in other words it's not your AI agent's fault. Your AI agent simply point out the ambiguity that already exists. So now we know where the root cause of the inconsistent is. The next step is identify where exactly they are. Who are they? And fix them. Surprisingly and not surprised the solution to identify the gray zone actually exists already in machine learning. And it's called active learning. So the idea of active learning is you have a lot of data point to be labeled. Imagine you'd have your initial model and you put it in production, maybe in the monitor mode. And you wanted to check the quality. But you don't have bandwidth to check all of them. Otherwise, it lose a point of having agent to do the work. But you still want to identify where the model didn't do well. And then active learning is to find a way to select the data points where tend to make mistake. And that's where it tells you where to pull your attention. So let's go over the traditional um machine learning models active learning pipeline and see how is different in the case of the new LM or AI agent error. So first, you train your initial model and then you make predictions on the unlabeled data set. It could be the production online data. And then from there, you see there's certain cases where the model wasn't sure because the probability is close to 2.5. And those data points are the high informational one that your model is going to learn the most. The uncertainty is a type of the signal to tell you where the model tend to make a mistake. Another popular type of way to figure out the problematic one is query by committee, meaning that it by a few different models or even same model multiple times and then you identify the disagreement. And that's where the model tend to make a mistake as well and that's where you need human clarification on label so the model can learn it from. The after you surface these problematic data points and for human to take a second look to make sure the label is actually correct or do we doesn't need additional information to dis ambiguate those cases meaning adding new features. And then you can add those label data set or potentially additional features to the next round of training. Then you retrain the model and then proceed uh continue this cycle. So then you can see in this particular active learning cycle you spend very little uh amount or at least in other way it's very efficient way to identify the problematic data points and spend your attention on those point where your model tend to learn the most. So that's the traditional one. The good news is the LM way is not very different. There's a major two part which we need some refinement. One is on this selection strategy. The other part is about retraining. On the selection strategy one um here we suggest you use the one about identify disagreement. At least from what we have tried so far the uncertainty score from LM is not very reliable. It's kind of a LM model doesn't know what it doesn't know. So when it's very confident it doesn't mean that it was the it would is correct. So that However, the the disagreement from different runs or from different models actually give you a more reliable signal where the model actually not sure about its verdict and it need human guidance. And then after you identify this group or the uh discriminant and need attention, you label them and provide a feedback like what you did before. Then afterwards, you might be thinking, "Oh, time to retrain the model." Yes or no. Yes, one option is retrain the model, but fine-tuning model is expensive. Here, we are proposing a more lighter weight solution, easier to iterate. It's about augment your agent with semantic and episodic memory. Let's use the previous example to illustrate what I mean here. Um as we talked about, in this case, the attempt to flip, it's mainly missing the information about you a particular company policy or preference. Here, we can add it to the um the knowledge base. If this customer complain about something outside hotel's control, then classify as positive because you don't want to pay attention to this. And then in the cybersecurity example, something similar, you can put in in that the password spray, meaning that the attacker trying to guess your password and try it with different times, even fail, and continuously. So, if without successful logging, then that should be benign. But, in contrast, if it succeed with final logging, then that should be malicious. So, here you actually identify the additional information to help disambiguate the two and then label them separately. And that's a clarity that help your model to learn. Also, help your human experts to label more consistently. So, this type of domain knowledge sharpen your decision boundary. And also, help the human beings. And this type of knowledge is like factual knowledge, is part of actually belong to your semantic memory. And this is in contrast with the episodic memory. Episodic memory is in the case where you haven't get around to distill the reasoning behind the earlier case why it should be on malicious or the benign side. Instead, you said that I have seen these similar cases before and it was labeled as benign. And let me leverage those past similar cases and they made a decision accordingly. And the advantage of the this episodic memory way is it's relatively automatically. It has less human intervention. I mean, the human intervention was done in the past already when during when you're doing a job online, um you can reference them, at least your AI agent can reference them automatically. They can making decisions without waiting for you to distill the reasoning and putting in the semantic knowledge. So, the idea is that the identify a similar case to the one under examination and then leverage their past decision as a reference. And then imagine there's a new case which is coming in the middle, doesn't belong to any of past group, and that's the one you still need a human attention. And this is how also um where episodic memory resolve recurring case automatically. And this particular useful in the case of cyber security alerts because a lot of noise especially those false positive noise are the one keep recurring. And that's the one are the low hanging fruit you can automate away. And with that human review which is a precious attention and bandwidth you have, now you can concentrate on those where episodic memory failed to address. And this is also how we chose between semantic memory and episodic memory. The two are actually not contradicting. In fact, they are complementary. Basically, you use episodic memory to address those recurring situations automatically. And then for what remains still reflecting after the reference from the past or no reference to be able to use, then you pass on to human review. And then human review distill the domain knowledge and put it in the semantic memory to be used next time. So here we talk about a solution actually brings three benefits. First, we identify these data point near the decision boundary and use the domain knowledge in semantic memory and this past similar case in the episodic memory, we're able to improve the the consistency significantly. So, now you have a more trustworthy AI agent for your user to use. And two other byproduct of this solution is you have a very efficient quality control. You don't need to check every agent output, but instead a subset of them, which your algorithm uh active learning think is potentially problematic, and you check them, either provide additional information or clarify the label, and your model will learn from there. So, most importantly, your labeling effort is very small, but high return. Last but not least, now you are gathering your customer feedback along this way. You not only have a trustworthy consistent model and high efficacy one, you also have one adapt to your customer environment. I'm pretty sure your customer will love your product if you your AI agent listen to them and adapt to them. So, here are some experiment results from uh real data points. So, here we collect uh 93 alerts, cybersecurity alerts, and then we run them three times. If without the solution we are proposing here, a quarter of them will flip-flop the verdict. In contrast, after applying our solution or using episodic memory, about 15% of them become consistent. However, there's still 10% of them remaining inconsistent. They still flip. Sometimes because there's no reference, similar cases in past paper used. Sometimes even there's a reference to be used, they still disagree after the second thought. But not a problem. The episodic memory do the automatic lock down all the inconsistency for 15%. The remaining 10% human review and provide the additional knowledge to disambiguate those tricky cases. After that, you also adapt to the customer environment that you want to learn. So, the three takeaways that I hope you have after this talk is number one, inconsistency isn't usually your model problem. Stop blaming your model, but instead focus your energy on label issue and potentially insufficient information and help your AI agent with this additional clarity. Second, the model disagreement isn't a bug, but feature. Treat each disagreement as an opportunity for your model to learn. Third, fine-tuning isn't your only option. Augment your agent with semantic and episodic memory. I hope you enjoy seeing your model keep improving and adapting to your customer environment. Last but not least, I want to thanks to my colleagues and wonderful friends at Data Dog. Special thanks to Ditun and Harsh, Anusha, and Stefan, Saeed, and Mati to give me a lot of enjoyable discussions and helped me to clarify some of the um the concept. And also, thank you for everyone in the Beats Secured Analyst team to make this journey enjoyable. And thank you for listening. Hope you will enjoy training your model and having it improving. Thank you. Bye.