Talk by Daniel Chalef (Zep AI) on provenance for knowledge graphs built by LLMs. He argues that because an LLM synthesizes facts from multiple sources, merges entities, and invalidates earlier facts, a simple source ID is insufficient; provenance itself must be modeled as a graph. In Graphiti, Zep's open source framework, sources are stored as episodes and each derived fact links back to them, so tracing a fact's origin is a graph walk. Tagging a source propagates metadata to all derived nodes and edges, enabling agents to filter facts by trust level (e.g., verified clinical sources). Deletion works as the same walk in reverse, supporting GDPR erasure while retaining facts still supported by other sources. The talk covers failure modes, merge/invalidation handling, metadata projection, mixed-trust sources, and benefits for compliance, veracity, and debuggability, followed by Q&A. # Observation Document: "Citation Needed: Provenance for LLM-Built Knowledge Graphs — Daniel Chalef, Zep AI" ## Explicit Self-Description - **Speaker identity and affiliation**: Daniel Chalef presents as a founder/team lead at **Zep AI**. Evidence: title credits "Daniel Chalef, Zep AI"; transcript: *\"My team and I built Graffiti… the open-source temporal graph framework and Zep, our enterprise agent memory infrastructure is built on graffiti.\"* - **Claimed scope**: The channel/speaker claims expertise in **LLM-based agent memory infrastructure**, specifically: - **Graffiti** — open-source temporal knowledge graph framework. - **Zep** — enterprise agent memory product built on Graffiti. - **Customer-base claim**: *\"Our customers derive context or agent memory for many user touch points… chat… voice transcripts, email, business data. And our customers have struggled with provenence.\"* - **Openness claim**: *\"Most of what I've covered today is in the graffiti framework… you can go to the graffiti repo on GitHub… try it out.\":** positions the open-source repo as accessible to the audience. ## Observable Presentation Patterns ### Intended Audience - **Technical/engineering audience** — evidenced by: - Assumed familiarity with LLM pipelines, knowledge graphs, BFS, sim-hash, deduplication, vector similarity search. - Conference-talk format with Q&A (*\"I don't know if we're doing Q&A here or outside\"*), slides, and QR codes. - Audience questions from practitioners (e.g., questions about graph mutation, edge types, file-based memory paradigms). ### Demonstrated Topics and Content Approach - **Core technical themes demonstrated**: - **Provenance/lineage** for LLM-synthesized facts — *\"tracing how an artifact was built and why.\"* - **Temporal knowledge graphs** as lineage-preserving data structures — *\"Lineage needs to be built in to the data structure… which is a graph, not logged afterwards.\"* - **Compliance use cases**: right-to-be-forgotten / retention-policy deletion (*\"A fact is only deleted if no remaining episodes support it\"*). - **Healthcare case study**: penicillin-allergy fact synthesized from EHR, lab PDF, and intake chat, used to illustrate veracity stakes (*\"could be a deadly mistake\"*). - **Presentation style observed**: - Scenario-driven explanations (healthcare patient example, "Daniel loves Adidas shoes" deduplication example in Q&A). - Admits cost/latency tradeoffs: *\"lineage and provenence is expensive. Graph construction is really expensive… we've put significant effort into reducing cost and latency.\":** - Interactive, candid Q&A responses, including acknowledging limitations (*\"not all of the provenence is in the graph\"*). ### Positioning Evidence - **Enterprise positioning**: contrasts his approach with file/markdown-based memory — *\"they just break down with the types of enterprise problems that we're solving, in particular provenance.\"* - **Product differentiation framing**: positions graph-native lineage as superior to source-ID pointers (*\"can't we just store like a source ID on the fact?… with context pipelines run by LLMs, this breaks\"*). ## Claims vs. Demonstrated Patterns — Separation | Creator Claim | Demonstrated in Material | |---|---| | Graffiti is open-source; Zep is enterprise infrastructure | Talk invites audience to GitHub repo; QR codes shown | | Customers struggle with provenance | Illustrative scenarios given; no customer data shown (appropriately abstracted) | | Lineage must be engineered into the data structure | Argued via failure-mode walkthroughs and graph-walk examples | | Cost/latency optimizations exist | Deferred to Q&A (*\"I'd be happy to speak to how we've done that\"*); specific methods only partially described | ## Notable Absences - No explicit channel-level self-description (e.g., "this channel is about X") appears in the transcript; identity is inferred from the talk format and product naming. - No marketing metrics, subscriber counts, or channel mission statements present. # Delivery Observations — Daniel Chalef, "Citation Needed: Provenance for LLM-Built Knowledge Graphs" ## Overall Tone and Register - **Conversational, engineer-to-engineer delivery**: The talk reads as an informal technical session, with filler words and self-corrections throughout (e.g., > "So, LLMs are really great at pulling together data from many sources. Uh, but they do so non sorry, they do so non-deterministically."). - **Casual, direct address to the audience**: He anticipates audience skepticism: > "So, I can imagine you're probably thinking, um, but can't we just store like a source ID on the fact?" - Spoken contractions and informal verbs recur: "you can zap the QR codes," "I'll give you the very high level outline for you." ## Vocabulary and Framing Habits - **Central term repetition**: "Provenance" (delivered as "provenence") and "lineage" are the anchoring vocabulary, restated across many contexts — compliance, debugging, deletion: > "So lineage needs to be an evolving set and survive mutation." > "So the rule is pretty simple here and it's easier to apply because the link exists." - **Consistent fact-centric framing**: He repeatedly decomposes claims into "fact + sources": > "Where did this fact come from? What is the veracity of this fact?" > "You can verify a fact based on its sources so you understand veracity. Should I trust this fact?" - **Technical shorthand and casual coinage**: "dduplication," "deconliction," "nasty gram," "zap the QR code" — informal labels for technical operations. ## Teaching Habits - **Concrete recurring example (healthcare/patient)**: A single stylized scenario (penicillin allergy from three sources) is carried through most of the talk: > "So here's an stylized failure mode. An agent retrieves context about a patient." It is reused for verification policies, tag inheritance, and deletion rules. - **Contrasting paired examples to illustrate policy differences**: > "So the facts are very similar shapes. They have three three parent episodes but opposite policies." (allergy flag vs. consent fact). - **"So" as a discourse marker**: Sentences very frequently open with "So" — observable dozens of times across both talk and Q&A ("So sets of links between facts…", "So to sum it all up…", "So in graffiti you can search…"). - **Explicit summary section**: > "So to sum it all up, deriving context is lossy and generative." — with a bullet-like list of benefits afterward (compliance, veracity, debugging, deletion). ## Communication Patterns in Q&A - **Restates questions indirectly and answers directly**, often first with a short confirmation: > "Yeah yeah so uh yes episodes are an entity on the graph." - **Uses mini-narratives to explain mechanics**: > "So Daniel loves Adidas shoes. Three months later, Daniel's shoes fell apart… We need to invalidate the Daniel loves Adidas shoes fact." - **Hedges and qualifies scope of answers**: > "not all of the um provenence is in the graph," "partly in graffiti partly in zap." - **Checks comprehension with the questioner**: > "Does that answer your question?" - **Practical cost candidness**: volunteers an aside on expense: > "as an aside, lineage and provenence is expensive. Graph construction is really expensive." ## Stable Patterns vs. Isolated Moments - **Stable**: "So"-initiated sentences, the healthcare example thread, fact/veracity/lineage vocabulary, informal register with fillers, rhetorical question framing. - **Isolated**: the "nasty gram"/Adidas anecdote (single use, Q&A only); the [snorts] moment at [528] (one-off, no pattern inferable from a single transcript); the "zap the QR code" phrasing (used twice near the close only). - **Uncertainty note**: Filler-word frequency and pacing are hard to quantify precisely from a transcript alone; timestamps bracket-line breaks, not speech pauses, so delivery-speed observations are not supportable here. # Citation Needed: Provenance for LLM-Built Knowledge Graphs — Daniel Chalef, Zep AI ## Position Summary Daniel Chalef argues that **provenance** (tracing how an artifact was built and why) must be engineered directly into the data structure — a knowledge graph — rather than logged after the fact, because LLM synthesis is non-deterministic and destroys the paper trail of derived facts. ## Core Claims and Reasoning ### The Problem: LLM Synthesis Destroys Lineage - LLMs pull data from many sources **non-deterministically**, interpreting and synthesizing summaries, extracted facts, and structured records. - Output artifacts **may not appear verbatim** in source inputs; synthesis "destroys the paper trail." - Provenance is needed for: - **Legal compliance** - **Debugging** - Deciding which sources to trust and which artifacts to delete - Solving this at scale is "a real engineering challenge." ### Stylized Failure Mode (Healthcare) - An agent synthesizes "Patient has a penicillin allergy" from three sources: an **EHR record**, a **PDF lab report**, and a **patient-typed AI intake chat**. - If the agent presents this fact to a doctor without indicating the source was the patient themselves, it "may mislead the doctor." - Claim: For complex agent applications, the ability to point to the exact source and its veracity is **often no**. ### Why Simple Source IDs Fail - Works in structured data warehouses/lakes where pipelines are deterministic. - Breaks with LLM context pipelines because: - Many facts are each synthesized from one or more sources ("J. Smith" and "John Smith" merge into one entity). - New data **invalidates old facts**; the store keeps changing "underneath your pointer." - Append-only logs become very hard to manage at scale. - Conclusion: **"Lineage needs to be an evolving set and survive mutation."** ### Graffiti's Graph Model - Sets of links between facts and sources are modeled as **graph relationships**: "Provenance in a context store containing facts is a knowledge graph." - Facts are graph triples (two entities + edge) hydrated from source **episodes**. - Tracing a fact to its source is "just a graph walk." - Challenges acknowledged: when entities merge, the merged entity must keep **all** source links or lineage is silently dropped; contradictions mutate edges, with an **invalid date** recorded and mutating source episodes noted against the fact. ### Metadata Projection / Tagging - Episodes tagged at ingestion (e.g., **EHR tag**) — derived entities and facts **inherit** the tag. - One tagging action supports filtering by veracity during graph walks. - Policy is situational, not baked in: for an allergy flag (life-and-death), one unverified parent **should block** a prescription; for "consent on file," **every parent** must be verified — facts of similar shape carry **opposite policies**. Chalef states: "your agent needs to execute or apply your business rules... It's situational." ### Deletion and Privacy Compliance - Retention policies and **right-to-be-forgotten** requests require lineage. - Rule: "A fact is only deleted if no remaining episodes support it" — the allergy fact survives deletion of the intake chat (two parents remain), while the contact-preference fact is deleted (solely derived from the deleted source). ### Conclusions - Deriving context is **lossy and generative**. - Lineage must be **built into the data structure**, "not logged afterwards." - Graffiti keeps sources verbatim and links everything derived back to them. - Benefits: compliance built in, veracity verification, debuggability, deletion decisions. - Qualification/asides: **"Lineage and provenance is expensive"** — significant engineering effort went into reducing graph construction cost and latency (details deferred to Q&A). ## Q&A Positions - **Edge relevancy changes / mutations (50 edges):** Some tracing is built into **Zep, not Graffiti**, using a separate data structure from the graph — "not all of the provenance is in the graph." - **Graph search:** Graffiti supports vector similarity search, full-text search, and relational operations (e.g., BFS) depending on the underlying graph database; provenance is available from any node you hit. - **Episodes and ingestion pipeline:** Episodes are nodes on the graph; derived artifacts are also nodes "because they too need to have lineage." Ingestion runs a pipeline: structured extraction (entities, relationships, candidate facts as subject–verb–object triples), then deduplication and **deconfliction** (example: "Daniel loves Adidas shoes" invalidated after a return and complaint). LLMs are used, but Chalef states they "try very hard **not** to use LLMs" where traditional IR/NLP, entropy, and sim-hash dedup are "far cheaper, far faster, more deterministic." - **File-based/markdown memory:** Chalef claims markdown "suffers from provenance" — mutating file lines makes lineage hard to track; file-based memory works well for desktop/single-user scenarios but **breaks down at enterprise scale**, particularly for provenance in multi-agent, multi-user, multi-source settings. - **Fact creation:** A **single-shot LLM extraction** extracts entities, relationships, and facts cheaply; a **reflection step** checks retrieval accuracy and enriches lineage ("why did it change," not just "this was related") — partly in Graffiti, partly in Zep. ## Unresolved Tensions - Graph-based provenance is claimed as the solution, yet some provenance (edge relevancy tracing) lives **outside** the graph in Zep. - Business-rule policies (verification thresholds) are explicitly **not** encoded in the graph, leaving correctness to the consuming agent. - Cost/latency of graph construction is acknowledged as a significant drawback, with mitigation details not fully presented in the talk. # Speaker Attribution & Provenance Record ## Video **Title:** Citation Needed: Provenance for LLM-Built Knowledge Graphs — Daniel Chalef, Zep AI **Video ID:** `01a15ca8-64a2-70fb-acd9-e9cdd5765f5b` ## Speakers & Roles ### Primary Speaker — Creator / Presenter - **Daniel Chalef**, Zep AI (identified in title metadata) - Role: **Presenter/creator** of the talk; sole presenter during the main session - Self-identifies as leading "my team and I" that built **Graffiti** (open-source temporal graph framework) and **Zep** (enterprise agent memory infrastructure) - Also answers audience questions in the Q&A segment ### Q&A Moderator / Host - **Unnamed voice** — opens Q&A ("Anybody have a question?", "Yeah, one right from the front", "one more") - Role: **Host/moderator**; manages question flow ### Audience Questioners (unnamed, 4 questions) 1. Question on mutating graph edges / relevance weighting 2. Question on edge types, entities, and graph search/resolution 3. Question on temporal support and how `add episode` works under the hood (LLMs, edges) 4. Question on file-based memory / markdown wikis paradigm 5. Question on fact creation — explicit vs. implicit, LLM summarization, which component (Zep or Graffiti) All audience questions are answered by Chalef. ## Promotional Segments — Flagged - **Product promotion:** Repeated promotion of **Zep** (commercial, "enterprise agent memory infrastructure") and **Graffiti** (open source) - **CTA:** "you can go to the graffiti repo on GitHub and I have a little QR code... you can zap" - **Closing CTA:** "if you'd like to learn a little bit more about Zep or Graffiti, you can zap the QR codes. Zep is on the left and Graffiti on the right." - **Customer references:** "Our customers derive context... Our customers have struggled with provenance" — unverified third-party claim about customer experiences ## Referenced Sources & Claims (per transcript only) - **Graffiti** — open-source temporal graph framework, hosted on GitHub (no URL given in transcript) - **Zep** — enterprise product; some provenance features (relevance tracing, reflection) exist only in Zep, not Graffiti - **Illustrative healthcare scenario** — patient penicillin allergy synthesized from three sources (EHR record, PDF lab report, AI intake chat) — **stylized example, not a real case** - **Illustrative example** — "Daniel loves Adidas shoes" used to explain fact invalidation - No academic citations or external documents are referenced ## Provenance Boundaries - Speaker attributions for Q&A voices are **inferred from conversational flow** (`>>` markers); individual questioners are not named - Transcript contains no timestamps - No curated metadata block was supplied beyond the title/speaker identification ## Comment Availability - **Not indicated** in the supplied source material So, LLMs are really great at pulling together data from many sources. Uh, but they do so non sorry, they do so non-deterministically. They interpret and synthesize data generating a summary, an extracted fact, uh a structured record. And this output artifact may not appear verbatim in the source inputs. Synthesis often destroys the paper trail of how these outputs were originated. And I'm going to be talking today about provenence, which is tracing how an artifact was built and why. Legal compliance often demands provenence, but it's also useful for debugging. Deciding which sources you trust and which artifacts to delete. And solving this at scale presents a real engineering challenge. If I can get my next slide going here. So my team and I built Graffiti uh the open-source temporal graph framework and Zep, our enterprise agent memory infrastructure is built on graffiti. Our customers derive context or agent memory for many user touch points. Those could be chat, but not only chat. Often it's voice transcripts, email, business data. And our customers have struggled with provenence. Where did this fact come from? What is the veracity of this fact? And over the next few slides, I'll share how we engineered solutions to this problem. So here's an stylized failure mode. An agent retrieves context about a patient. So this is a healthcare scenario. And what comes back is a clean confident fact. Patient has a penicellin allergy. And the context was synthesized from three sources. A lengthy EHR record, electronic health record, a PDF lab report, and something a patient typed into an AI intake chat. If the agent presents the fact to a doctor in treat in a treatment scenario without clearly indicating the source was from the patient themselves, it may mislead the doctor. When an agent retrieves context, can we point to the exact source and its veracity? For complex agent applications, the answer is often no. So, I can imagine you're probably thinking, um, but can't we just store like a source ID on the fact? This can work well in structured data warehouses or data lakes. A pipeline outputs one value copied or mutated deterministically, and the sources are known and easily marked. But with context pipelines run by LLMs, this breaks in several ways. You prompt an LLM with several sources. Many facts are each synthesized from one or more of the sources. Somebody like J. Smith and John Smith are merged into a single entity, one identity. And John's facts are derived from many different places. So new data might invalidate old facts. The store keeps changing underneath your pointer and an appendon log which often might you know might come to mind here gets very hard to manage at scale as there's so many changes occurring. So lineage needs to be an evolving set and survive mutation. So sets of links between facts and their sources can be modeled on a graph as relationships. So provenence in a context store containing facts is a knowledge graph. We have three in this example we have three source data in graffiti that turned episodes. We have two entities extracted from the episodes patient and penicellin and an edge between them. This graph triple the two entities and the edge can be hydrated as a fact. Patient has a penicellin allergy. Tracing a fact to its source is just a graph walk. So it's pretty simple and easy to map source to fact on the first right but keeping it correct while the graph changes can be really hard when new data uh so for example when two entities merge the merged entity needs to keep all source links from both otherwise we silently drop a source and we lose lineage. And when new data contradicts existing data, mutating it, we need to capture this lineage too. In the rightmost card, a fact is rendered invalid by new data. And in graffiti, an invalid date is added to the mutated edge. And the source episodes that resulted in the edge mutating are noted against the fact. So gaining graffiti, the relationship between source data and derived artifacts such as facts is easily modeled on the graph. With metadata projection, we can also model classifications that span many different episodes. and facts derived from them. And so I'll give you an example here. In in the prior healthc care scenario, episodes may originate from an EHR record and have an EHR tag associate associated with them, but not all records are. And so on ingestion, we tag the episodes with the EHR tag. All subsequent entities and facts derived from the episode inherit the tag. And so if the agent wants to retrieve only facts from verified clinical sources, it's very simple to filter for the appropriate tag as we walk the graph. So one tagging action at ingestion supports evaluating the veracity of a fact. But what if the fact is three parents or more? Here we have a verified flag as our metadata and in this case one parent is not verified. So is the fact verified for the allergy flag which could be a life and death situation. the agent missing it, missing that particular flag could be a deadly mistake. So not retrieving the fact and any source of the three should block that prescription being issued. But for something like a consent on file for procedure fact, the mistake is operating on unverified consent. So the patient hasn't actually given their consent and every parent needs to be verified. So every single episode should have that tag. So the facts are very similar shapes. They have three three parent episodes but opposite policies. And here graffiti or the underlying store exposes that choice. It exposes which of the episodes have the gra the particular tag, but your agent needs to execute or apply your business rules. So that's not necessarily something that we bake into the graph. \[snorts\] It's situational. Another situation where lineage is really important, we may have to delete source data due to retention policies or right to be forgotten requests. So privacy compliance and this is really tricky because if we have context derived from multiple sources, how do we do so? Mapping lineage here is really useful. We know which facts are derived from the source data we intend on deleting. But what if only some of the source data needs to be deleted but not all? So in this example, we need to delete the intake chart data. So what the patient filled in, which is only one of three source data. In graffiti's model, the allergy fact survives the deletion and that's because there are two parents parent episode still supporting the fact. However, the contact preference fact is deleted as it was derived solely from the deleted source data. So, the rule is pretty simple here and it's easier to apply because the link exists. A fact is only deleted if no remaining episodes support it. So to sum it all up, deriving context is lossy and generative. Lineage needs to be built in to the data structure, engineered into the data structure, which is a graph, not logged afterwards. And in graffiti we keep the sources verbatim and we link everything back everything derived from those sources back to the source. And provenence offers many benefits to users of graffiti. You have compliance built in which makes your chief compliance officer very happy. You can verify a fact based on its sources so you understand veracity. Should I trust this fact? It's easy to debug where something came from. So, why do I have this fact? How was it generated? And also determining what to delete. And most of what I've covered today is in the graffiti framework. So, you can go to uh the graffiti repo on GitHub and I have a little uh QR code QR code later that you can zap um and try it out. So, by the way, as an aside, lineage and provenence is expensive. Graph construction is really expensive uh in the way that graffiti does it. And so we've put significant effort into reducing cost and latency of generating graph artifacts. And I'd be happy to speak to how we've done that uh in in the Q&A. So thanks for attending. Um if you'd like to learn a little bit more about Zep or Graffiti, you can zap the QR codes. Uh Zep is on the left and graffiti on the right. And I don't know if we're doing Q&A here or outside. Okay, happy to do Q&A. \>\> Just repeat the question. \>\> Yeah, \>\> we have time. \>\> Anybody have a question? \>\> Yeah, one right from the front. \>\> How do we mutate the graph at the edge? \>\> Oh, to account for weight changes in relevancy. Um that is some structure that we've actually built into Zap not into graffiti and um what we do is we do have for that a some tracing that we do which is kind of x of the graph. So not all of the um provenence is in the graph. Sorry, you \>\> if you have 50 edges, \>\> uh, sorry, is was that a question? If you have 50. Oh, there you go. \>\> I said I I use 50. \>\> You use 50. \>\> Yeah, \>\> 50 edges. Um, so so you know, Zep is able to uh look at provenence across those but using a separate data structure from the graph. \>\> Okay. Thank you. Thank you \>\> for that particular problem. \>\> So uh does the agent also create the edges edge types and the entities itself or how does it resolve those edges types? \>\> So in graffiti you can search across the entire graph. Uh it has um vector similarity search against various textual artifacts um full text search as well as graph relational operations things like BFS um it depends on the underlying uh graph database that's used and so your agent can walk the graph it can search semantically etc. And obviously from anywhere you hit in the graph you're then able to understand the provenence of a particular artifact that you've hit. Uh I find it very uh fascinating this temporal support out right out of the gravity database. Can you uh tell just a little more um under the hood what are those really the the episodes are nodes in the graph just like the other nodes and how do you extract so I saw the API that let's say I give some uh ad ed episode how do we extract information from the ad episode call under the hood like large language models and and edges \>\> yeah yeah so uh yes episodes are an entity on the graph uh or a node on the graph. Um it makes sense to model them that way. Um in Zep and graffiti we have various derived artifacts that are um nodes on the graph as well because they too need to have lineage and we need to understand how they were derived. Um, and in terms of how the ad episode method works, there's a pretty complicated uh, pipeline that gets run on um, uh, episode ingestion and I'll just give it the very high level uh, uh, outline for you. So there's a structured extraction extracting entities and the relationships between them and candidate facts and those are the materialized or hydrated fact triples. So two entities and a fact. Um and it's structured as uh a fact is structured as um um uh subject well subject uh verb object. Um and after that there is a uh deconliction process that runs dduplication and deconliction process. We dduplicate entities and we deconlict facts because there might be existing facts in the graph that are going to be mutated by a new learned fact. So Daniel loves Adidas shoes. Three months later, Daniel's shoes fell apart. He sends it back to those the shoes back to the return application and he sends a nasty gram along with it. We now Daniel returned shoes as a fact and Daniel was unhappy about Adidas. We need to invalidate the Daniel loves Adidas shoes fact and so that is part of that pipeline as well. A lot of what we do uses LLMs but we try very hard not to use LLMs in this process as well. So where we're able to deploy more traditional information retrieval techniques, more traditional NLP techniques, uh looking at things like entropy and a bunch of other, you know, using um sim hash and a bunch of other approaches to ddupe, uh we do so uh far cheaper, far faster, deter more more far more deterministic. So hopefully that answer your questions. \>\> Yep. Uh thank you for the great talk. Um just wanted to ask a question. So it seems like a common theme these days in uh memory systems is more filebased memory and wikis and knowledge bases and uh I'm just wondering have is Zep working on something like that and also uh could some of the ideas here be represented in that paradigm? \>\> Yeah. uh markdown suffers from provenence. File-based um file-based uh memory starts to break down with provenence. It's very difficult when you m mutate lines in a file to understand the lineage or the provenence of why those changes occurred. Um, not only that, but in multi- aent, multi-user, and multi-source scenarios, it can be very challenging to manage markdown files at scale. I think they work really well um for desktop usage. Uh, they sometimes work well in uh agentic use cases that are server based, not necessarily desktop or single user, single agent scenarios. Um but what we found is that it's um they just break down with the types of enterprise problems that we solving in particularly in particular provenence as an example. Does that answer your question? Oh more I don't know how much time we have left but maybe one more \>\> uh Daniel thank you. Um question on the right bag. How do you do that explicitly or implicitly? Um, how do you create the facts? Do you ask the LLM to summarize the conversation or at every turn you do that? And and which component does it, Zep or? \>\> Uh, \>\> yeah. So, so we do um as part we actually as part of the extraction, we've managed to get um a singleshot extraction working that extracts entities and the relationships between them and facts. and we're able to do so really cheaply as a consequence. Um, and so yes, we're using an LLM for that. Uh, we do have a reflection step or some reflection built in to ensure that the things that we've retrieved um are actually accurate um as well as to do some other stuff around uh uh more more richness to lineage. So why did something change? Not just this was related but also why did it change. Uh that's uh partly in graffiti partly in zap. Yeah. All right. Well, thank you everybody.