Talk by Peter Werry of Unblocked on moving beyond similarity-search RAG to a relational context engine. The speaker argues that engineering context such as pull requests, reviews, tickets, wikis, and chat has structure (authors, dates, states, labels, relationships), so retrieval also needs natural language translated into a database query, with the model writing the query rather than answering directly. The workshop builds a query engine over a GitHub repository through six stacked pull requests. The description names four components for reliable queries: schema discovery by sampling documents into a merged schema, identity resolution mapping a person to their GitHub login with fuzzy matching, query planning with a forced name-extraction tool call and the current date, and deny-by-default validation with an operator allow list and tenant isolation. It also says a retry loop feeds errors back to the model, showing where full text search helps when exact match fails, and that the same engine powers a chat agent. # Video Identity Observations ## Source Context - **Title:** *Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked* - **Format:** A condensed one-hour technical workshop/live coding session (originally two hours), with Q&A at the end. ## Intended Audience The video demonstrates a **practitioner/developer audience**, specifically engineers building AI retrieval or agent systems: - Attendees are expected to follow along hands-on with a laptop: *"here's the QR code for setup"* and *"you probably should have run this ingestion thing."* - Assumes fluency with developer tooling: *"a series of six pull requests and they stack on top of each other,"* *"prereqs that you'll need to run through."* - Q&A questions come from people thinking about production concerns — *"how does that work at scale… what's the cost?"* — indicating a professional engineering audience. ## Claimed Scope - **Explicit thesis:** *"rag… is only half of retrieval and today we're going to build the other half which is turning natural language into real database queries."* - **Concrete deliverable:** something *"capable of filtering, providing aggregations, it's time scoped, has full text search… it's validated, it self-corrects, and it powers a full-blown chat agent."* - Framed as transferable: *"you could take this into your organization and just deploy,"* though with the caveat, *"I would recommend maybe working a little bit more on the validation harnesses before deploying this internally."* ## Self-Description Evidence - **Speaker identity:** *"I'm Peter. I'm from Unblocked… my colleague Brandon."* - **Company self-description:** *"We build a context engine. Our company's name is Unblocked… we deliver organizational context to humans and increasingly of course agents."* - **Positioning on RAG:** treats semantic similarity search as *"a good jump off point"* but insufficient alone: *"there are questions that rag can't touch. All those structural questions around things like state or temporality."* - **Production practice vs. workshop technique:** *"at unblocked we're using a lot of static schemas because we want to get the correctness absolutely nailed down,"* while the dynamic schema discovery shown is presented as *"a good technique"* for exploration. ## Observable Presentation Patterns - **Workshop-style delivery:** stepwise build through PRs, live demos of queries (*"open PRs by Peter and bam"*), and reminders that *"the repo's up online… you can run through it at your leisure."* - **Engineering-value framing:** repeated emphasis on procedural code over LLM nondeterminism — *"I wouldn't recommend using LLMs for this part"* (validation) and *"you always need some kind of structural harness around it."* - **Event presence signals:** mention of lingering after the talk and being *"at the P16 booth… all week,"* indicating conference-based public engagement. - **Domain focus in Q&A:** names top data sources — *"Slack is the most important for why questions"* and PRs contain *"all of that like organizational history"* — revealing where the channel/organization centers its claims about organizational context. ## Explicit vs. Demonstrated - **Creator claims:** Unblocked's product scope, continuous webhook-based ingestion, "best practice memories" distilled from PR comments, and use of hybrid search ("The answer is yes"). - **Material demonstrates:** the hands-on construction of a schema-aware, validated, self-correcting natural-language query engine over a document store, addressed to working engineers at a technical conference. # Delivery Observations: Peter Werry, *Beyond RAG: Build a Relational Context Engine from Scratch* ## Scope and Evidence Limits - **Source:** Title and auto-style transcript only. No curated metadata was supplied. - **Not observable:** Vocal tone, volume, pacing in seconds, facial expression, and slide content. Observations about pace and tone are inferred only from wording and transcript structure. - **Speaker attribution:** The transcript does not label speakers. The opening is clearly the presenter (Peter), but later passages mixing "I" and "we" and audience Q&A are less certain to be Peter alone. - **Transcription noise:** Some words are garbled (e.g., "PRexs," "Jura," "schemalis," "Levenstein," "Rashine"). Quotations below are reproduced as they appear. --- ## 1. Recurring Verbal Fillers and Hedges The most frequent non-content words are **"um"**, **"uh"**, and **"so"**, which appear throughout the talk. - Opening: "Okay. So, I'm I'm Peter. I'm from Unblocked. Um, over here is my colleague Brandon." - Mid-talk: "Um so the thesis here is" - Repeated hedging qualifiers: **"basically,"** **"kind of,"** **"sort of,"** **"pretty,"** and **"arguably"** > "it's arguably sort of the simplest part." > "it's fairly non-deterministic." > "The error loop is actually one of the most important things." **Observation:** Hedges appear often in descriptive passages, while firm statements are used for recommendations and warnings (see Section 4). --- ## 2. Recurring Tag Questions and Check-Ins The speaker frequently ends statements with a confirming tag: > "that's just basically black hole, right?" > "So you want you want that like conceptual matching, right?" > "Does that make sense?" > "Hopefully people can see that. Great." **Observation:** The presenter checks audience comprehension repeatedly, most often with "right?" and "Does that make sense?" This appears in at least three distinct places, which suggests a consistent habit rather than a one-off. --- ## 3. Signposting and Scaffolding Language A strong pattern is explicit narration of structure: announcing what is coming, referring back, and naming steps. **Forward signposting:** > "So this is what we're going to do to start with." > "we're going to be following a series of six pull requests and they stack on top of each other." > "We'll come back to that in a sec." **Back-referencing:** > "as I said um at the beginning like there's similarity search is actually really powerful for jump off points." > "the thing that I referred to a few slides ago about identity resolution." **Step labeling:** > "the second step is..." / "Now the last part is..." / "So now I'm going to write run another query" **Summary and recap:** > "So just a quick summary of what we built." > "So just to recount on this here are the reusable ideas here. Schema aware prompting dynamic schema discovery identity resolution validation and error retries." **Observation:** Signposting is the most consistent structural habit in the transcript. Each major section is introduced, named, and later recapped. --- ## 4. Direct Recommendations and Cautions The speaker often gives explicit, first-person opinions about what to do or avoid. These are stated more firmly than the hedges in Section 1. > "I don't recommend that. you should probably separate your collections by tenant" > "I wouldn't recommend using LLMs for this part." > "I would recommend maybe working a little bit more on the validation harnesses before deploying this internally" > "I would suggest using procedural tricks to to do this type of thing" **Observation:** Recommendations use **"I would recommend"** or **"I wouldn't recommend"** consistently, distinguishing advice from description. Firm language is reserved for these moments. --- ## 5. Contrast Framing (Naive vs. Reliable) A recurring teaching technique is to set up a naive approach, then show why it fails. > "the instinct even two years ago was let's just dump this all in a vector store and retrieve it and rerank it and hopefully we get the right thing. Um, but that's that's not going to work." > "it would be naive to say that generating the query is the only thing that we have to do." > "So what you could end up with is the wrong answer where you pull up PR that's semantically related but it was merged three years ago." **Observation:** The presenter frequently poses a simple approach and then complicates it. This is used as the central teaching device for motivating each new step. --- ## 6. Concrete Example Habit Abstract concepts are routinely anchored to a specific query or scenario: > "show me Peter's open PRs" > "PRs I merged last week" > "did we work on anything authentication related last week." **Observation:** Examples are reused across the talk as running threads (the "Peter's PRs" and "authentication" queries recur), which helps connect separate sections. --- ## 7. Live-Demo Narration The presenter narrates the demo as it happens, including interruptions and recoveries. **Acknowledging failure or delay:** > "Okay. So, um, I'm just going to repeat..." / "Hopefully that renders. No, it won't. Of course. Give me one second to switch." > "And hopefully people can see this." / "Whoops." > "I apologize. That's not going to be wired up just yet." **Reading from the screen:** > "You have to forgive me. I actually have to read this off of the screen out here." **Celebratory or completion markers:** > "Bam. There we go." > "Perfect." > "That's it." **Observation:** Demo narration is frequent and candid. The presenter names problems openly rather than moving past them silently. Completion is marked with short, informal exclamations. --- ## 8. Audience Engagement and Q&A Handling **Pre-question engagement:** > "Did Did everyone uh was everyone able to actually set it up? Just curious. No. Okay." > "Anyone else? I thought I saw a question over there. Oh, yep." **Repeating questions for the room:** > "I'm just going to repeat the question so everyone can can hear it." **Requesting clarification before answering:** > "Now that that's a very overloaded term, so I'll just make sure I understand exactly what you're talking about." > "let me just make sure I understand your last question." **Deferring to later:** > "If you want uh you can come and fetch Brandon and I afterwards" **Observation:** The presenter consistently repeats questions for the audience and asks for clarification before answering. This is a clear, repeated practice. --- ## 9. Pronoun Shifts: "I" vs. "We" The transcript shows a pattern of switching between personal and collective pronouns: - **"I"** for personal actions and demo steps: "I'm going to step through," "I just want to talk," "I'm gonna now skip." - **"We"** for product and team decisions: "we build a context engine," "at unblocked we we create these things called best practice memories," "we control the schema actually." **Observation:** "We" is used when describing company practice, while "I" is used for demonstrative actions and personal recommendations. This is a visible pattern, though the transcript doesn't clarify whether "we" also refers to Brandon. --- ## 10. Domain Vocabulary and Terminology Key terms are introduced and then reused consistently: - **Core framing:** "context engine," "organizational context," "semantic," "structural," "temporal" - **Retrieval:** "retrieval has two halves here semantic and again today we're going to be talking about this structured queries." - **Safety:** "deny by default," "allow list," "untrusted output," "prompt injection point" - **Method:** "schema aware prompting," "identity resolution," "error loop," "procedural" **Observation:** Terms are defined in context and reused, which supports consistency across sections. The speaker also clarifies overloaded terms when audience questions use them (e.g., "hybrid search"). --- ## 11. Pace and Compression Signals The presenter explicitly names pacing choices: > "It's it's now one hour so it's quite condensed. U which means that I'm going to step through the workshop pretty quickly." > "We're going to skip through this." > "Okay, let's move." **Observation:** The presenter tells the audience when material is being skipped and encourages later self-study ("the repos up online um and you can run through it at your leisure"). This is explicit and repeated. --- ## 12. Honest Uncertainty and Limits The presenter acknowledges limitations of the approach: > "this is obviously not going to work." > "it is fairly non-deterministic. So you always need some kind of structural harness around it" > "at least like this iteration of that algorithm is pretty hacky, right?" > "You're always actually gonna you're going to miss something." **Observation:** Limitations are stated directly and not minimized. This is a consistent part of the delivery across Q&A and the main talk. --- ## Summary | Feature | Consistency in Transcript | Notes | |---|---|---| | Fillers ("um," "uh," "so") | High | Very frequent throughout | | Hedges ("basically," "kind of," "pretty") | High | Used in descriptions | | Tag questions ("right?", "Does that make sense?") | Moderate to high | At least 3–4 instances | | Signposting and recaps | High | Structural backbone | | Explicit recommendations ("I would recommend") | Moderate | Used for cautions and advice | | Naive-then-complicate framing | Moderate to high | Used to motivate each step | | Concrete running examples | High | "Peter's PRs," "authentication" | | Candid demo narration | High | Includes apologies and recoveries | | Repeating questions for the room | High | Consistent Q&A practice | | "I" vs. "We" shift | Moderate | Attribution uncertain | | Named pacing and skipping | Moderate | Explicitly stated | | Stated limitations | Moderate to high | Across talk and Q&A | **Caveat:** These observations are based on a single auto-style transcript with no timing or audio information. Patterns labeled "high" appear multiple times in the text, but the sample is one talk, so they describe this delivery only and should not be generalized to the speaker's other presentations. # Observation Record: "Beyond RAG: Build a Relational Context Engine from Scratch" — Peter Werry, Unblocked ## Session Overview - **Speaker:** Peter Werry (Unblocked), with colleague **Brandon** present but not speaking substantively in the transcript. - **Format:** Condensed one-hour workshop (originally two hours), walking through six stacked pull requests building a natural-language-to-database-query engine. - **Stated setup:** "We build a context engine. Our company's name is Unblocked. And we deliver organizational context to humans and increasingly of course agents." ## Core Thesis > "Rag... is only half of retrieval and today we're going to build the other half which is turning natural language into real database queries." - **Definition qualifier (explicit):** In this talk, "RAG" means specifically *semantic similarity search or vector retrieval* — not retrieval generally. - **Central paradigm shift:** "The model doesn't actually answer the question in this paradigm. It writes the query that accesses the database." ## Positions and Reasoning ### 1. Context is scattered and has both semantic and structural dimensions - Organizational context lives in PRs, review comments, issues/tickets (Linear, Jira), docs/wikis (Notion, Confluence), and conversations (Slack, Teams). - This data is **semantically queryable** but also has **structure**: authors, dates, states, labels, relationships. - His characterization of the older instinct: "even two years ago... let's just dump this all in a vector store and retrieve it and rerank it and hopefully we get the right thing... but that's not going to work." ### 2. RAG fails on structural/temporal questions - Worked example: the query **"PRs I merged last week."** Under RAG, you embed the sentence, find nearest vectors, rerank top-K — and may get PRs *semantically about* merging from **three years ago**, because RAG cannot enforce author, merge state, and temporal relationships. - Qualification offered: RAG still shines for **content questions** (find the code, the PR that changed it, the Slack thread that debated it), and as a "semantic head start" jump-off point before an agent recurses the file system. ### 3. Four requirements for reliable query generation 1. **Schema awareness** — the LLM must know the schema before writing queries. 2. **Identity resolution** — "show me Peter's open PRs" won't work if the GitHub login differs from "Peter"; the system must map human names to login names. 3. **Security/validation** — untrusted query output could execute JavaScript server-side (e.g., via `$where`/function operators), making it "a prompt injection point." 4. **Self-correction** — errors on first attempt should feed back into the LLM for retries. ### 4. Implementation steps demonstrated - **Schema discovery (procedural, not LLM-based):** Sample 300 documents, collapse into a single tree, skip nulls, infer types. Enums detected via "procedural tricks" — character-count heuristics (2–8 characters for type info), pattern matching. He framed this as a deliberate choice: LLM extraction "works but it's fairly non-deterministic," so you need "some kind of structural harness... running procedural code." - **Identity resolution:** Simple fuzzy matching (Levenshtein distance). He acknowledged this "seems like we're coupling to the schema a little bit" but said it is "absolutely possible to determine this dynamically" via LLM-synthesized mappings. - **Query planning (the LLM step):** Pass the schema, force a tool call for name extraction, include **current date/time in the prompt** because "LLMs are actually pretty good about generating temporal queries if you pass them the current date and time." - **Validation via allow lists, not deny lists:** "We don't deny list these things. We allow list them. So it's deny by default." Hidden metadata fields (e.g., tenant ID) should be hidden from the LLM and wiped from output, with the system injecting its own metadata. - **Retry loop:** Failures can come from validation or execution (e.g., mixed inclusion/exclusion); errors are passed back to the LLM "up to some number of max attempts." He called the error loop "actually one of the most important things." - **Full text search:** Added after the retry demo showed exact-match failing on "authentication" once regex and FTS were disallowed. ### 5. Recommended practice for deployment - He cautioned the workshop build is not deployment-ready: "I would recommend maybe working a little bit more on the validation harnesses before deploying this internally." - Final system capabilities claimed: filtering, aggregations, time-scoping, full text search, validation, self-correction, powering a chat agent. ## Q&A Positions - **Scaling schema size:** Add another LLM step that selects the right schema from small summaries/descriptions per query — "LLMs are surprisingly good at this. They weren't a year ago, but now they're very good at it." - **Ingestion at scale/latency:** At Unblocked, content is continuously ingested via webhook callbacks on every PR event — framed as "a distributed systems question." - **Top data sources:** **Slack** is "the most important for why questions" (conversational memory, e.g., "roughly three weeks ago, I was having a conversation with so-and-so"); **PR comments** contain organizational history and are mined — e.g., Unblocked creates "best practice memories" distilled from past PR comments. - **MCP exposure:** A structured query table "could express that through MCP," even dynamically updating the tool as schema changes. - **RAG vs. structured queries — explicitly complementary:** "No, not at all. No, they're absolutely complimentary." Semantic search handles conceptual matching that textual/FTS cannot; his recommended pattern is vector search as a starting point, then structured-query tools. - **Static vs. dynamic schemas in production:** "At Unblocked we're using a lot of static schemas because we want to get the correctness absolutely nailed down." Dynamic discovery was presented as a technique for data exploration, experiments, or arbitrary databases. - **Sample-size tuning for schema discovery:** Derived "empiric... trial and error" through customer feedback loops ("you missed the thing that I was searching for"); he conceded "you're always actually going to miss something." - **Forcing tool calls:** A validator watches the agent; if name extraction isn't called, "you just basically insert a user message that says you forgot to call the extraction tool." - **Schema changes over time:** Unblocked's schema is static and tied to their codebase; if GitHub changed its schema, they'd adopt changes statically and deploy via CI. For dynamic discovery, rerun schema generation on a sampling interval. ## Tensions and Qualifications Preserved - **Contradiction acknowledged by the speaker:** The talk promotes dynamic schema discovery, yet Unblocked itself uses **static schemas** in production for correctness — he framed the workshop technique as better suited to exploration and arbitrary databases. - **Repeated procedural-over-LLM stance:** LLMs are used only for query synthesis and retry reasoning; extraction, validation, and sanitization are deliberately procedural, on non-determinism grounds. - **Tentative framing kept intact:** Recommendations like tenant separation ("I don't recommend that you should probably separate your collections by tenant") were stated as advice, not settled fact; the demo repo he called "pretty hacky" in its current iteration. # Speaker Attribution and Provenance: Beyond RAG Workshop **Video ID:** 01a15ca6-acad-7d61-9092-fd13a0b9b8de **Title:** Beyond RAG: Build a Relational Context Engine from Scratch, Peter Werry, Unblocked --- ## 1. Speaker Inventory ### 1.1 Identified Speakers - **Peter Werry** - **Role:** Presenter and primary speaker - **Affiliation (stated):** Unblocked - **Activity:** Delivers the workshop, runs the demos, and answers audience questions. - **Identification basis:** Self-introduction ("I'm Peter. I'm from Unblocked.") and title attribution. ### 1.2 Referenced but Not Confirmed as Speaking - **Brandon** - **Stated role:** "my colleague," introduced alongside Peter. - **Speaking status:** The transcript does not attribute any utterance to Brandon. Peter refers to him in the third person ("come and fetch Brandon and I afterwards"). - **Uncertainty:** His presentation role is not explicitly stated. Whether he spoke is unknown from this transcript. ### 1.3 Unidentified Speakers - **Audience members** asking questions during the Q&A segment. - Their names and affiliations are not given. - Their questions are only partly captured, and several turns are marked only by speaker-change markers. - Peter's responses are the only attributable content in that segment. ### 1.4 Other Named Individuals (Not Speakers) - **"Rashine" / "Rashab"** - Appear in an illustrative example: "my colleague... his name is Rashine. His user login name is Rashab." - Presented as example data, not as participants in the session. - The names may be transcription artifacts. The transcript does not confirm the spelling. --- ## 2. Promotional Segments The following segments are **flagged as promotional** based on their content. They promote Unblocked or the company's booth and products. > **Promotional segment flags** - **Company description (opening)** - "Our company's name is Unblocked. And uh we deliver organizational context to humans and increasingly of course agents." - **Product self-reference (within Q&A)** - Peter refers to "unblocked" as a system that users can query ("say Oh yeah, you know, roughly three weeks ago..."). - Peter describes Unblocked's internal features: "at unblocked we we create these things called best practice memories." - Peter describes Unblocked's ingestion model: "unblocked um at unblocked like we're continuously ingesting content." - **Event booth invitation (closing Q&A)** - References a "P16 booth," with activity "starts later today at around four o'clock and we'll be there all week." - **Product-adjacent recommendation** - "you could take this into your organization and just deploy." (Accompanied by a hedge: "I would recommend maybe working a little bit more on the validation harnesses before deploying this internally.") --- ## 3. Referenced Sources ### 3.1 Workshop Materials - **Setup QR code** displayed on screen for participants. - **Online code repository** for the workshop. Peter says "the repos up online." - **Series of six pull requests** that "stack on top of each other." These are referenced throughout the workshop but are not shown in the transcript. - **Keynote slides** used during the presentation. ### 3.2 Platforms and Tools Named as Data Sources - **GitHub** (pull requests, user logins, merge states) - **Linear** and **Jira** (issues and tickets) - **Notion**, **Confluence** (documents and wikis) - **Slack** and **Teams** (conversations) - **PostgreSQL** ("PG"), mentioned as a comparison for schema reading - **Model Context Protocol (MCP)**, mentioned as an integration option ### 3.3 Techniques and Named Methods - **Levenshtein distance**, named as a fuzzy-matching technique for identity resolution. - **Full-text search (FTS)** and **hybrid search** (semantic similarity plus lexical search with rank fusion). Peter confirms his company uses hybrid search. - **Vector retrieval** and **reranking** (described in the rag framing). ### 3.4 Customer Feedback - Peter states the sampling approach was refined through "trial and iteration with customers," who reported missed documents. The transcript does not provide specifics. --- ## 4. Third-Party Claims and Attributed Assertions These are claims made by the speaker about outside parties or about general practice. They are recorded as attributed statements, not verified facts. - **About GitHub's schema stability:** Peter says that if GitHub changed its schema, "that of course rarely happens because that would have serious implications for their customers." - **About LLM capability:** Peter says LLMs "weren't [good at this] a year ago, but now they're very good at it" (referring to selecting relevant schemas). - **About prior industry practice:** Peter describes the approach of "dump this all in a vector store" as the instinct "even two years ago," which he says "is not going to work." - **About the source of best-practice memories:** Peter says these are "distilled from PR comments in the past." --- ## 5. Comment and Metadata Availability - **Curated metadata:** Only the title is supplied. No speaker list, description, chapter markers, or citation metadata are present. - **Comment availability:** No comment data is supplied with this record. Comment availability is **unknown**. - **Transcript quality:** The transcript includes several probable transcription artifacts, including "Levenstein" (likely Levenshtein), "jura" (likely Jira), "Rashine/Rashab," "PG," and "P16 booth." It also contains audio-noise markers and unlabeled speaker changes during Q&A. --- ## 6. Provenance Limits - **Unavailable channels:** - The six pull requests referenced in the workshop are not provided. - Slides and screen content are not available. Peter describes several on-screen states that the transcript cannot show (for example, the "payoff" query results and the chat tab output). - Audience questions are partly inaudible or unlabeled. - **Unverified identifiers:** Speaker names are taken from self-introduction and title. Brandon's role and speaking status are unconfirmed. - **Demo outputs:** Results Peter describes from live demonstrations (for example, "bam, there we go") are speaker reports, not independently verifiable records. Okay. So, I'm I'm Peter. I'm from Unblocked. Um, over here is my colleague Brandon. And today, uh, well, first of all, I just want to talk a little bit about what we do. So, uh, we build a context engine. Our company's name is Unblocked. And uh we deliver organizational context to humans and increasingly of course agents. Um so the session was originally a couple hours. It's it's now one hour so it's quite condensed. U which means that I'm going to step through the workshop pretty quickly. If you don't get uh you know all the steps nailed down as we go don't worry about it because the repos up online um and you can run through it at your leisure um afterwards. Um, so the thesis here is that rag and I'm just going to say that rag in our parlance today is going to be semantic similarity search or vector retrieval. Okay, rag is only half of retrieval and today we're going to build the other half which is turning na natural language into real database queries. So here's the again the QR code for setup um if people haven't already done so. I'm gonna now skip um on to the next thing and I'm just going to briefly bring up hopefully that renders. No, it won't. Of course. Give me one second to switch. Whoops. And hopefully people can see this. Uh this is the repo here. So setup steps are pretty simple. Um there are a few sort of uh dependencies up here, prerexs that that uh you'll need to run through. Um and the the course throughout the course of the workshop, we're going to be running through a series of pull requests uh and the network is basically gone or at least isn't running properly. Let's try this one instead. Well, the good thing is we don't need network for this talk. Um, so hopefully you got everything before we started. Um, because this one's obviously not going to work. But at any rate, uh, we're going to be following a series of six pull requests and they stack on top of each other. So, I'll be There we go. I'll be referencing these as we go. Um, and I'll just go back to here so that everyone can see that. And back to the keynote. Now I'm gonna have to extend it again. Sorry. \>\> There we go. All right. Hopefully people can see that. Great. So um our agents need context and context is scattered across every tool that engineering teams live in. So you're going to find it in PRs, review comments, um issues and tickets like in linear and jura docs \[clears throat\] and wikis, notion confluence, and of course in conversations and places like Slack and Teams. And all of that is semantically queryable, but it's it also has structure. It has authors, dates, uh, states, labels, relationships, etc. And you know, the the instinct I I love to say this like even two years ago is ancient history in the realm of AI, but the instinct even two years ago was let's just dump this all in a vector store and retrieve it and rerank it and hopefully we get the right thing. Um, but that's that's not going to work. So I'm just going to step back and and kind of relate this to the larger discussion about context. um in in the context of a context engine that serves organizational context um this is this is made up of more than just code as I said we're connecting everything together so the scattered context uh on the left in all your code and PRs docs etc goes in and the idea is to get you know grounded context that relates to each other out the other side that's specifically tailored to the task at hand um and in that model uh we have to care not just about the semantic relationships, but of course the structural relationships that connect all the data. So again, rag is about embedding the query, finding docs with similar meaning, and then we take that stuff and we stuff it into the prompt, right? Well, um, that shines for content questions like find the code, find the PR that changed it, find the Slack thread that debated it, but um, and if you wrap it in an agent loop, you can get pretty far with that. And you can also wire this thing up to like um, you know, tools that can recurse files and it acts as a good jump off point. you get a good semantic head start and then from there your agent can recurse over the actual file system in GitHub um and you can get pretty far with that. But in our experience um our this is not really the way things go in practice. engineers often ask questions like this PRs that I merged last week or my team merged last week. Um they're they're basically, you know, hunting for something in particular and they know where the starting point is, but they don't know where the destination is, but they do know the relationships. So, as an example, PRs I merged last week, let's think about this in a rag context. you'd you'd you know embed that sentence, you'd find the nearest vectors, you'd rerank it top K. Uh you get PRs that mention merging about a talk last week, but uh you wouldn't get the actual relationships like the author, the merge state, um and whether it was merged last week actually. So what you could end up with is the wrong answer where you pull up PR that's semantically related but it was merged three years ago. Um and so that doesn't actually answer the question because there's a structural there's a temporal aspect to that question. And so retrieval has two halves here semantic and again today we're going to be talking about this structured queries. And here's the key. We're going to take uh the model doesn't actually answer the question in this paradigm. It writes the query that accesses the database. Okay. So, um it would be naive to say that generating the query is the only thing that we have to do. Um in order to generate the query and to do it reliably um and and to take care of maybe even some security concerns, we need to look at all four of these points here. So that first of all the LM needs to be aware of what the schema is before uh creating the query. It needs to understand uh some of the relationships in the data. So for example, if you pass a natural language query that says show me Peter's open PRs, um then it's not going to know what to do because it'll synthesize a query that says Peter when really the GitHub login name might be different. So it needs to be able to map that across. Um, and of course you're going to have some uh untrusted queries coming out the other end. Um, maybe using the function operator in the in parlance which might execute JavaScript directly on the server and we probably don't want that. Um, and then finally, if there are errors the first time around, we want to be able to self-correct those things. So here's here's what this looks like in one single shot. What we want is something that looks like this. Um it it maps uh my colleagueu's name is Rashine. His user login name is is Rashab. Um and then we want to be able to map things like the state and get maybe uh operators like limit and sort. So uh this is what we're going to build. We're going to start with a full UX, a stubbed engine, um, that is not capable to start with, and we're going to gradually build this up. And then at step three, the thing will come alive. And so we'll actually be able to issue queries to it and see what it does. And uh, if you hadn't seen that before, now is another good time to scan that and get it in. I'm here to help you. \>\> Yeah. \>\> Yes. Any repo is fine. Doesn't matter which one. Um and if if you want to take a shortcut, you can just point an agent at the thing and let it set everything up for you. That works. Okay, let's move. So, this is what we're going to do to start with. Um, by now you you probably should have run this ingestion thing. Don't worry if you haven't. Um, again, like you can revisit this at the end. I'm just going to quickly show what the the kind of barebones interface looks like for this thing. So, now this thing is running. Now, if I type anything in here, this is obviously not going to work. Um, and what you should see is something like that. Um, there's also a tab for chat. This is a fully implemented agent uh that can uh do work for you. So, we'll come back to that in a sec. And so, this this is the contract in the code if you wanted to read. Um, we're going to don't worry about this. We're going to skip through this. Um, but basically like that's the entry point and we're going to be building around that. So the first thing we want to talk about is schema discovery. Um during ingestion, if you if you look at the way that the ingestion script worked, it it actually had to build out a schema in order to download the content and store it. Um but you can imagine that you may not know the schema. So the ingestion may be happening in a service elsewhere and you want to build like a fully dynamic schema discovery engine. Um schemalis. So there's no schema to read. um unlike you know PG or something where we could read the table definitions. Um so what we'll do is we'll sample the documents to infer the schema and that's the whole trick. So the we're just going to reconstruct it. We're going to take um all the nodes and merge them down. So this is going to sample 300 documents, collapse it into a single tree. uh nulls are skipped because missing fields won't be able to constrain the type. And then uh we're going to do something kind of interesting here with enum types. And so now uh with with enum types, we can do an interesting sort of procedural trick. This is just one thing. I would suggest using procedural tricks to to do this type of thing because you could pass it through an LLM and do extraction and that works u but it's fairly non-deterministic. So you always need some kind of structural um harness around it that is running procedural code to either validate the things that you pull out or to actually do the extraction itself. And in this sample project um it's the procedural code that's doing the extraction. And let's go and show that. So we'll do and then run it again. And now what we get is this thing here. So the schema ran uh the the discovery process ran and it created a schema for us that outlines what the pull request schema looks like in GitHub. It even created some indexes for us. Now, this is like marked down on purpose because we're going to take this schema and pass it to the LLM when we synthesize the actual uh query. Now, the second step is the thing that I referred to a few slides ago about identity resolution. And this is because like my lo GitHub login um is similar but it's not quite the same. And so if I were to ask a question like um hey find all the PRs by Peter um it if without doing some sort of identity mapping it would just say you know it would just match across Peter and it's not going to find anything. So we have to be able to solve that. \[snorts\] And again, um, we're going to do this. At first blush, this seems like we're coupling to the schema a little bit here. Um, but you know, it is absolutely possible to determine this dynamically. And you can have uh your LLM essentially synthesize the mappings for you. So, but in in our case, we're just going to do these simple little tricks. Um very simple sort of fuzzy matching. Um Levenstein distance is fine. You can just like match across the names and uh there you go. Oh, actually I want to show you one thing before we get on to that one. So let's do the same thing again. We can pop up \>\> Oh yeah, I'm sorry. Absolutely. Tell me when it's big. Good. Thanks. Good call out. Okay. So, let's run our thing again. And now, um Oh, actually, I have to I have to wait till the next step. I apologize. That's not going to be wired up just yet. So, we'll go on to the thing that will really give us the payoff here, which is step three. So this is the actual query plan step and it's it's it's arguably sort of the simplest part. It's the LM part. Um and what we do here is we pass the schema give it a tool and force it to call that tool for name extraction and synthesis. Um and then it will execute the plan. There's one other thing which is the temporal aspect. uh LLMs are actually pretty good about generating temporal queries if you pass them the current date and time. So that's another important thing to do in the in the prompt. And so here this is the name extraction part. We're going to force that tool call to happen in this case. And at this point I am going to show you the the payoff for this. So there we go. So now I can do things like this. open PRS by Peter and bam. So what we'll see here is that it uh generated this um this match clause and uh you can't see it from here but it it mapped Peter to my GitHub user login puery and generated the query and that's the outcome. So that's cool, but um we need to get on to the next step because even though we are able now to query the data, we have some safety concerns. Um it could be the case that it produces untrusted output and so you know it could use the wear or function clauses and literally run JavaScript on the server which means that this is now a prompt injection point. Um, you could also have hidden fields in your schema like ID or metadata fields that leak internals. So, we need to sanitize um, and we don't deny list these things. We allow list them. So, it's deny by default. You want to create basically an allow list of allowed operators and uh, you know, potentially an allow list of allowed fields. In our case, this is dynamic schema discovery, so we're not going to do that. Um, but we could have some fields that we know about like metadata fields. And this is a a useful thing for like tenant uh tenant isolation. So if you have a a uh collection that houses multiple tenants, I don't recommend that. you should probably separate your collections by tenant, but if you did, then you could have a metadata field that contains the tenant ID and you would want to hide that no that um information from the LLM, but also validate the output. So if it outputs that metadata field, you need to wipe it out and then you need to inject your own metadata field underneath. And again, like technically this is it's not actually that difficult to do. You just have to write the code. It's all procedural. Um I I wouldn't recommend using LLMs for this part. And so again, allow list, don't deny or deny everything, I should say. Now the last part is um we're going to this kind of dovetales into the next step. At the end of validation we're obviously going to return an error and it's going to get dumped out. And right now that error is just it's just there. We're not going to do anything with it just yet. But I will show you now what this looks like. Um, oops. Stay on that one and run our service. So now I'm going to write run another query that checks for authentication in the title and we've disallowed regular expressions. So now what we can see is that it rejects the query. And finally, this is the thing that kind of ties it all together is the retry loop. The plan can fail in a couple of different ways, right? Um, it can fail validation. So maybe the uh the LM actually outputs things that are illegal according to our validator. Or it could actually fail during execution. Um, so for example, let's imagine that it it mixed an inclusion exclusion project. Um, then that's obviously going to fail. And then but we can self-correct. We can take that error and pass it back into the LLM and say hey that was wrong. Um maybe you can do better up to some number of max attempts of course. So I'm going to show now what this looks like back over to our project so we can punch up a level. Whoops. Sorry. You have to forgive me. I actually have to read this off of the screen out here. Okay. So now if we run that same query. Perfect. So, what we can see is that it took several attempts to get it right. Um, by the third attempt though, there's a bit of a problem here because we said no regular expressions, no FTS. Um, so now it said we're going to just use match, but now it's an exact match. So it tries to find authentication fails which means there's one more thing that we can do which is this. We'll add full text search. It's pretty basic. Um we need to create an index obviously and we'll combine it with author and state filters uh in the same match. And so now we should be able to answer a question like this. And let's see what happens. And see if it is actually going to do what I said it's going to do. Bam. There we go. And so now the the final payoff here is if we go into the chat tab, we can ask some sophistic more sophisticated questions that um you know an agent can actually reason about behind the scenes. So you can say things like um did we work on anything authentication related last week. So here we can see that it generated the query. Now it's pulling the data and kind of reasoning about it. There we go. That's it. So, just a quick summary of what we built. We're basically like reaching the end. There's going to leave lots of time for questions. So just to recap here, um there are questions that that rag can't touch. All those structural questions around things like state or temporality. Um you don't get the right data out when you ask those types of questions with rag, traditional rag. So the same engine behind the query playground and the chat agent, uh they're they're the same. So if you guys want to play with that after, go ahead. Um it's it's the same backing thing. The only difference is of course that uh the chat agent actually does reasoning whereas the query interface is just for queries. So just to to recount on this here are the reusable ideas here. Schema aware prompting dynamic schema discovery identity resolution validation and error retries. The error loop is actually one of the most important things. And so we built something that's capable of filtering, providing aggregations, it's time scoped, has full text search. Um, it's validated, it self-corrects, and it powers a full-blown chat agent. So you could take this into your organization and just deploy. I I would recommend maybe working a little bit more on the validation harnesses before deploying this internally, but um yeah, it'll it'll answer all your questions about your PRs. And that's it. So, we've left lots of time for questions now. \[laughter\] Did Did everyone uh was everyone able to actually set it up? Just curious. No. Okay. If if you want uh you can come and fetch Brandon and I afterwards, we'll just kind of linger outside here. Oh, at the and at the P16 booth as well. Yeah, but that that starts later today at around four o'clock and we'll be there all week. \>\> Cool. Any questions? \>\> Yes, \>\> I have a question. I think you did a great job breaking down like basics. \>\> Yeah, \>\> a lot of my mind goes like okay so at scale how does for example you you generate the schema and you put it into the system prompt how does that work at scale? It's like also depending on the size of your company, how are you fitting all this data? How do you choose what data and like how long does it take to process all this? What's the cost? That's where my mind is. \>\> Yeah, it's a good question. So, you know, um I'm just going to repeat the question so everyone can can hear it. The demo showed obviously a fairly contrived example where you're just focusing on pull requests. Um and so the question is how do you scale this? Like how do you do this across multiple different collections, multiple different tables, etc. Um, so you you can add uh another LM step on top where uh this the schema collections have tiny little summaries and descriptions and then uh depending on what the query is, you can load the right schema for that query. LMS are surprisingly good at this. They weren't a year ago, but now they're very good at it. Um, so that's kind of the way to shave the context thing down. And that's to your question about scaling things out. Um, was there another aspect of scaling that you were interested in? \>\> Getting all the data in and I guess like part of that is like cost. \>\> Yeah, it's livveness and stuff too. \>\> How long does it take to do like what is what data should I pick to put into this? \>\> Yeah, it's it's a good question. So unblocked um at unblocked like we're continuously ingesting content. Uh we get web hook callbacks for all the events for example in in PRs. And so every time there's a change, we get that web hbook event, we ingest the content, we rem um and store it in the database. Um that that's just really just like a distributed systems question about how you scale that out from an ingestion level. Um you know, storage, same thing. So yeah, \>\> so maybe this will make it more concrete. So what if you would say like these are the top three data sources like someone seems like GitHub maybe like something \>\> um so Slack for sure a lot of the like why questions and and decision- making happens in Slack um people uh in the context of this sort of like structural query um people usually think about conversations that they've had in the past um and they'll be able to query unblocked and say Oh yeah, you know, roughly three weeks ago, I was having a conversation with so- and so. And so that that's a perfect thing for structural query. Um yeah, like I would say that um Slack is the most important for why questions. Um a lot of the why decision-m is also embedded in PRs. PRs uh have all of that like organizational history in the comments and that's just basically black hole, right? So uh you can basically run over that data before you serve it and and kind of mine it for interesting insights. So for example at unblocked we we create these things called best practice memories which are distilled from PR comments in the past. Does that make sense? \>\> Yeah. Yep. Any else? Yeah. \>\> Question. directly into how this building. \>\> Yeah, I mean you you can definitely do that. So if you have a structured if you have a structured query table, you could express that through MCP. You can even do that dynamically by updating the tool every time the schema changes behind the scenes. Um this this type of thing is showing like what happens on the other end of that MCP call. So you you could you know have an MCP interface that just takes natural language queries for example and then translate that all on the back end or you could provide a direct interface to the database. It depends on like what the customer interface looks like, but yeah. Yeah. \>\> Yeah. \>\> Yeah. \>\> Yeah. Yep. Second part is better. Thoughts around \>\> uh how to mean how to combine it or let me just make sure I understand your last question. \>\> Yeah. feel like this is like \>\> No, not at all. No, no, they're absolutely complimentary. Yeah. Yeah. I mean, you can have a full text search that does this kind of um you know, like that's full text search, right? So, it'll identify keywords. You still need some kind of mechanism to to do like the actual semantic similarity search because there you all have ideas and sentences that are encoded that aren't not ne necessarily textual matches. So you want you want that like conceptual matching, right? So as I said um at the beginning like there's similarity search is actually really powerful for jump off points. So you can for example start with a vector search and then from there give it access to tools like this because then it'll actually have like some interesting starting data and help it form queries to go and and fetch the data. So yeah that that's that's something that we do and so sorry the first part of your question was \>\> so you take um you take you know in in anyways like these are all documents and they're they kind of like form into JSON so you can merge those things down and look for certain tokens and then you you know, extract the tokens and then that's how you distill the types. So you you can look through the source code and you'll see that at least like this iteration of that algorithm is pretty hacky, right? It'll do things like look to make sure that the character count is between two and eight for uh for type information and then for enumerations it's it's looking for particular patterns as well. \>\> Yeah. \>\> Yeah. Yes. \>\> Yeah. \>\> Now, that that's a very overloaded term, so I'll just make sure I understand exactly what you're talking about. Do you mean hybrid search in terms of um a provider that does semantic similarity and then also does a lexical search and then does rank fusion? Yeah. Oh, yeah. Yeah. No. So, so we we do that. Yes. The answer is yes. Yeah. \>\> Yeah. \>\> Yeah. \>\> Uh just empiric empirical trial and error. And so I mean there's there's no easy solution to this. Um be behind the scenes like at unblocks we're we're using a lot of static schemas because we want to get the correctness absolutely nailed down. This this was like a just a technique that I wanted to show because um you know for exploration of data or experimental um uh for experiments or either to wire up like arbitrary databases this is a good technique. So we we got this through some trial and iteration with customers and you know we we'd ingest a whole bunch of contents from the document store and we'd get responses back from customers that said oh you missed the thing that I was searching for and we'd dig into it and we'd figure out why and we kind of got to the sample size that made the most sense. Yeah, \>\> it's absolutely true. Yeah, definitely. You're always actually gonna you're going to miss something. There's always going to be some document that has a little edgy thing. Yeah. Anyone else? I thought I saw a question over there. Oh, yep. Sorry, can you say that first part again? \>\> There's mention that you understand the schema. \>\> Uh yeah, what we do is we force a tool to be called for not for schema discovery because that happens at the start. It's for name extraction. we so we have a validator that watches what the agent's doing. So like if it doesn't call the the first part which is name extraction. Um so there's there's actually two techniques for this. One is you create a tool that does the the query and as an argument to the tool you accept a username. So you can fail right there if the username is not correct or malformed. Um, if you want to do name extraction uh via tool call, then you have to watch what the agent does. And then if it doesn't call it, you just basically insert a user message that says you forgot to call the extraction tool. Do so now. Yeah. \>\> Yeah. How do you handle the changes? Yeah. \>\> Yeah. So remember like for this particular example the schema is is completely unknown and so as documents get added um that schema could update. So you your your choice is either you know on a sampling interval you you rerun the schema generation um in our case at unblock the schema is is static and it's tied to our codebase and we control the schema actually. So if the schema of if GitHub decides to change the schema um that of course rarely happens because that would have serious implications for their customers but if they did then we would just adopt whatever changes are made statically and then obviously deploy that through CI. Yeah. Anyone else? \>\> Awesome. I think we can uh reclaim some time in that case. Thanks. \>\> \[music\]