Kuba Rogut (Turbopuffer) presents a 50-task benchmark (ContextBench) evaluating semantic code retrieval on Claude Code. The benchmark measured whether the agent found the right files, lines, and symbols rather than whether it solved the task. Three conditions were tested: raw Claude Code, windowed grep reads capped at 50 lines, and windowed reads plus a Turbopuffer-backed semantic search tool. Results cited: raw Claude Code wastes one in three file reads, windowed grep reduces this to one in five, and adding semantic search reduces it to one in eight, with file precision rising from 65% to 87%. Semantic search performed best on behavior-adjacent tasks where files share no keywords, while grep won on import tracing. Cursor's production figures are cited (24% relative improvement in answer accuracy, 2.6% increase in code retention); the speaker attributes his smaller gains to Claude Code treating semantic search as just another tool rather than knowing when and why to call it. # Observation Document: Benchmarking semantic code retrieval on Claude Code — Kuba Rogut, Turbopuffer ## Speaker & Explicit Self-Description - **Speaker identity**: "My name is Kuba. I'm here from Turbo Puffer." - **Company self-description (explicit claim)**: Turbopuffer is "a serverless full text and vector search database built from first principles on object storage." - **Claimed customer base**: "We serve some of the fastest growing AI companies in the world"; in Q&A, "we are the vector database that powers companies like Cursor, Anthropic, Notion." - **Speaker role**: A technical employee presenting engineering work ("we built a simple CLI tool… we call it Turbo Grep"), not stated job title. ## Intended Audience - **Developer/engineer audience** familiar with or curious about AI coding agents (Claude Code, Cursor), vector search, and code-retrieval benchmarks. Evidence: assumes familiarity with terms like "agentic search," "grepping," "chunk, embed and index," and references Cursor's blog posts and "Context-Bench" paper without explaining them in depth. - **Live talk audience** — talk format with closing Q&A: "If you have any questions, feel free to come up," followed by multiple audience questions with applause. ## Claimed Scope of the Presentation - Benchmarking semantic code retrieval as an add-on tool to Claude Code: - "Let's see for real… how much better does it work?" - Tested three conditions: "raw cloud code out of the box," Claude Code "with a max of 50 line reads at a time," and "windowed reads with the T Puffer search tool." - Demonstrates a tool built by Turbopuffer: "We built a simple CLI tool for cloud code. We call it Turbo Grep… chunk it, embed it using the Voyage code model and then upload it to Turbo Puffer." - Notes open-source status: "There is an open source library for the V1 version and then soon the V2 version will be open sourced." ## Observable Presentation Patterns - **Evidence-driven framing**: presents precision/recall metrics ("baseline it hit like 65% precision… semantic search got it to like 87% file precision") and human-readable translations ("one in every three file reads is actually just a completely wasted file"). - **Positions Turbopuffer technology favorably while acknowledging limits**: concedes Claude Code's default grep "actually does win the file recall" and that results "weren't as great as Cursor's because part of Claude code is it's built for just grepping." - **Forward-looking thesis tied to product category**: "The people that provide these easy tools to provide to shrink down these billion context windows into the right million will win in the long term." - **Engaged, technical Q&A handling**: answers on embedding model ("Voyage code three"), preprocessing ("just the raw code"), and use cases (multimodal data, knowledge bases like Notion). ## Claims vs. Demonstrated Content | Creator claim | Observable evidence | |---|---| | Turbopuffer powers Cursor, Anthropic, Notion | Stated assertion; no demonstration within this talk | | Semantic search boosts precision | Benchmark numbers (87% vs 65% file precision) presented on slides/transcript, but underlying dataset only described ("human labeled data set," "across 50 tasks") | | Tool "Turbo Grep" exists and works | Shown via "a little video of… a tool call trace for a cloud code" on the Django repo | ## Channel Presentation Notes - Format: conference-style technical talk with slides, a demo video, and live audience Q&A. - Tone: informal, first-person engineering presentation ("we built," "we tested") with company advocacy embedded ("if you'd like to know more about the talk or any question about Turbo Puffer… find me after"). - No explicit channel-level self-description (e.g., channel name, mission statement, upload cadence) appears in the transcript; metadata beyond the title was not supplied. # Delivery Observations: Kuba Rogut — "Benchmarking semantic code retrieval on Claude Code" ## Tone & Overall Style - **Casual, conversational tone** throughout: frequent filler phrases like "you know," "kind of," "like," and "I mean" appear constantly, e.g. "it kind of found agentic search which is actually you know, grepping through your file system kind of worked better." - **Self-aware and playful**, anticipating audience skepticism: > "what do you mean cache compute? This sounds like a bunch of baloney you're throwing at me." ## Vocabulary & Framing - Heavy use of **technical jargon** delivered accessibly: "agentic search," "chunk, embed it," "golden files," "precision," "recall," "parent-child relationship." - Repeatedly **translates metrics into plain English**: > "a more like English or like human version of it is that... one in every three file reads is actually just a completely wasted file." - Uses **"quote unquote"** as a verbal hedge: "a quote unquote wasted or irrelevant file," "where semantic search won you know, quote unquote won." ## Teaching Habits - **Explains by analogy and concrete numbers**: "if cloud read 10 files and eight of them were needed, it hits 80% precision." - **Pre-empts audience questions**: "You may ask me like why this like 50 line read thing limit? It's because it became really noisy really fast." - **Anticipates counterarguments**: "you may also think like these numbers seem kind of small like what 2.6% 2.2%? Well, keep in mind that..." - **Structure of "thesis → evidence → summary"**: explicitly recaps with "So in summary like what does this mean?" ## Recurring Phrases & Verbal Tics - **"kind of"** — extremely frequent modifier ("kind of worked better," "kind of a Cursor style," "it kind of shows up"). - **"you know"** — pervasive connective across nearly every paragraph. - **"essentially"** as an explainer lead-in: "It's essentially you can imagine it's just a simple way of using a tree splitter library." - **"I think" / "I believe" / "I can't speak for"** — hedged claims, especially about third parties: "I can't speak for how they do it, but I just know they do do it." ## Q&A Delivery - **Comfortable admitting limits**: "Oh, that's that's tough. I mean it's hard to say," and "In this case it was just simple just the raw code." - **Speculates transparently** when unsure: "I can imagine it's definitely something of providing not just code level meaning." - Brief, direct answers to factual questions (embedding model, vector DB), then elaboration. ## Uncertainty Notes - Word-level fillers ("like," "kind of") are consistent across the whole talk and Q&A — this is a stable pattern, not isolated. - Some transcript irregularities ("cloud code" for Claude Code, garbled numbers like "23 and 1/2 per uh percent") suggest transcription noise rather than speaker error; delivery judgments based on these should be treated cautiously. # Benchmarking Semantic Code Retrieval on Claude Code — Position Record **Speaker:** Kuba Rogut, Turbopuffer ## Background & Framing - **Claude Code by default does not use semantic code search.** Kuba cites a tweet from **Boris**, described as the "founding father of Claude Code," stating early versions used semantic search with a local vector DB, but **agentic search (grepping the file system) worked better and seemed simpler**. - **Cursor (a Turbopuffer customer — "one of our very first") does use semantic code search**, indexing codebases into Turbopuffer because they see real performance gains. - Cited from Cursor's blog: **~24% relative improvement in answer accuracy** for the Composer model; **~12.5–13% across all models** ("I think it's like 12 and a half or 13%" — hedged). - Online A/B test: **2.6% increase in code retention in large codebases**, **2.2% decrease in dissatisfied user requests**. - Kuba's qualification: these numbers look small because **not every query needs or benefits from semantic search** — Cursor states this in the blog post itself. ## Core Thesis: Embeddings as "Cached Compute" - Grep-based (agentic) search repeats grep → read → grep cycles **in every session and every agent**, so compute is redone even for identical questions. 6,000 tokens once is small, but "across every session and every agent this really starts to add up." - Semantic search has an **upfront cost** (chunk, embed, index) but then acts as a **cache of semantic meaning**, letting agents retrieve understanding faster with token savings. - He notes people now run multiple agents simultaneously ("I'm running like three at one time"), so long-term savings matter. ## TurboGrep Tool - A simple CLI tool for Claude Code: - Uses a **tree splitter library** to parse, chunk, and **embed via the Voyage code model** (specified in Q&A as **Voyage Code 3**). - Uploads to **Turbopuffer**; simple file system walkthrough. - V1 open-sourced; V2 "soon will be open sourced." - Demo: querying "password reset token generator" in the Django repo via the T-Puffer tool returned contents and a full explanation. ## Benchmark Methodology - Uses **context bench** (public paper; Cursor has an internal one): tests **whether the agent found the right files, lines, and symbols** during problem-solving, not just final task success — "the process really matters." - Human-labeled dataset of golden files/lines/symbols per task; **50 tasks** total. - **Three conditions:** 1. Raw Claude Code out of the box 2. Claude Code with a **max 50-line read limit** (windowed grep) 3. Windowed grep **plus T-Puffer semantic search tool** - Rationale for the 50-line limit: unlimited reads became "really noisy really fast" and diluted differences. ## Results ### Precision (fraction of files read that were golden files) | Condition | File precision | Human-readable framing (Kuba's) | |---|---|---| | Baseline | ~65% file, 33% line, 43% symbol | "one in every three file reads is completely wasted" | | Windowed grep | higher | one in five reads irrelevant | | Grep + semantic | **~87% file precision** | one in eight files irrelevant | - Kuba attributes low baseline precision to Claude Code's **exploratory default behavior** — "it loves to read as much as it can." ### Recall (fraction of needed files found) - **Baseline Claude Code wins file recall** — again because it explores every file it can. - Line recall drops a lot for baseline: it explored "the wrong things" — many files without golden context lines. - Windowed grep and grep + semantic had **roughly equal recall**, with semantic search showing **no improvement and even some decreases** — an acknowledged negative result. - Breakdown by task type revealed **stark differences**: - **Semantic search won** on tasks needing **behavior-adjacent files lacking shared keywords** (example: finding multiple ORM-handling implementations across libraries that keyword search missed). - **Grep won** on tasks that were **tracing imports**, where keywords appear early in tool calls. - Conclusion: "certain tasks require different types of tools" — two genuinely different file-finding methods. ## Why Gains Fell Short of Cursor's - Kuba's stated explanation: **Claude Code is built for grepping**; the semantic search tool was added as an extra tool ("here's this cool tool, you probably should use it sometimes"), and the model lacks true understanding of **when and why** to call it. - **Cursor's Composer treats semantic search as a built-in tool** it knows when and how to use — hence their larger gains. ## Summary Positions - Semantic search **boosted precision substantially**; grep and semantic search find code in **different ways**. - Long-term winners will **provide lightweight tools to find the right context in various ways**: "You can't just grep through everything unfortunately." - "The people that provide these easy tools to shrink down these billion context windows into the right million will win in the long term." ## Q&A Positions - **Definition of semantic search:** "just doing vector search" — embed query with Voyage Code 3, send to Turbopuffer. Turbopuffer "powers companies like Cursor, Anthropic, Notion." - **On messy/undocumented code:** works best when code has **good inline comments** — one repo performed well specifically because of comments above functions; "the embedding isn't the hard part, it's figuring out what meaning really is of that chunk." - **Preprocessing / parent-child chunking:** TurboGrep embedded **raw code only**, no parent-child structure. He speculates (explicitly hedged — "I can't speak for" customers) that Cursor uses its own embedding model and "I think they actually do what you said — they create fake comments on top of your code and then embed the code with the comments," enabling high recall. - **Role of vector DBs vs. grep long-term:** grep is **zero-cost** and fine locally; vector DBs are built for **multiplayer, complex data relationships, and multimodal data** — "you can't grep through a video file, an audio file, an image file." Whether a vector DB helps "depends on the workload." ## Preserved Tensions & Qualifications - **Negative result kept visible:** semantic search did not improve recall and caused some decreases on certain tasks. - Recall vs. precision **trade-off across conditions**: baseline maximizes file recall; semantic search maximizes precision. - Several claims are explicitly hedged: Cursor's Composer 2 gains ("you can probably imagine"), cross-model accuracy numbers, and internal details of Cursor's embedding approach. - Conflict between stated claims: Claude Code's creators found grep "worked better," while Cursor's data shows semantic search gains — Kuba reconciles this via tool integration depth rather than resolving it. # Speaker Attribution & Provenance Record **Video:** Benchmarking semantic code retrieval on Claude Code — Kuba Rogut, Turbopuffer **Video ID:** `01a15ca9-144c-75b9-8102-aa027000a5ee` ## Speakers | Speaker | Role | Evidence | |---|---|---| | **Kuba Rogut** | **Creator / presenter** — introduces himself: "My name is Kuba. I'm here from Turbo Puffer." Delivers the main talk. | Transcript opening | | **Unnamed audience members (multiple)** | **Guests / Q&A questioners** — several distinct questioners during post-talk Q&A (questions on definition of semantic search, performance on messy code, preprocessing/embedding strategy, vector DB long-term role). | Q&A section | | No separate **narrator**; no curated metadata supplied identifying additional speakers. | — | — | ## Provenance & Referenced Sources (per speaker) ### Attributed to Kuba Rogut (presenter) - **Tweet by "Boris"** — described by Rogut as "the founding father of cloud code," stating early Claude Code versions used semantic search with a local vector DB before switching to agentic (grep-based) search. *(Third-party claim cited in talk.)* - **Cursor blog posts** — on semantic search and on indexing code bases; cited figures: - ~24% relative improvement in answer accuracy for the Composer model; ~12.5–13% across all models. - Online A/B test: 2.6% code retention increase in large code bases; 2.2% decrease in dissatisfied user requests. - **ContextBench public paper / benchmark** — human-labeled dataset testing whether agents found required files, lines, and symbols; also noted Cursor has its own internal context bench. Rogut ran 50 tasks across three conditions (raw Claude Code, 50-line windowed reads, windowed + Turbo Grep). - **Turbo Grep tool** — presenter's own CLI tool (tree splitter library, Voyage Code 3 embedding model, uploads to Turbopuffer); V1 open source, V2 "will be open sourced" (stated as future fact, not confirmed done). - **Turbo Grep benchmark results** — presenter's own measurements (e.g., baseline 65% file precision → 87% with windowed grep + semantic search; one-in-three vs one-in-eight wasted file reads). - **Third-party claims about customers** — states Turbopuffer is "the vector database that powers companies like Cursor, Anthropic, Notion" and that Cursor uses its own embedding model and reportedly "injects" fake comments before embedding (**explicitly hedged**: "I can't speak for how they do it, but I just know they do do it"). ### Attributed to Q&A questioners - Definition request; question on messy/undocumented code; question on preprocessing (parent-child embeddings); question on vector DB long-term role. No independent sources cited by questioners. ## Promotional Segments - **Flagged:** Opening self-introduction of Turbopuffer ("serverless full text and vector search database… we serve some of the fastest growing AI companies"). - **Flagged:** Q&A closing promotion — "we are the vector database that powers companies like Cursor, Anthropic, Notion. So when you use something like Cursor you have by proxy used Turbo puffer." - **Flagged (implicit):** The talk presents Turbo Grep and its benchmark results favorably; promotion of the tool/company is interwoven throughout (recorded, not evaluated). ## Availability & Uncertainty Notes - **No curated metadata** was supplied; no comment availability could be established. - Numeric claims from Cursor's blog and the A/B test are **second-hand** (quoted by Rogut, not shown). - Presenter's speculations (e.g., Composer 2 gains, Cursor's fake-comment embedding practice, multimodal customer workloads) are **explicitly marked as beliefs/imagined**, not verified facts. - Audience speakers are **unidentified by name** in the transcript. \>\> Hi everyone. Welcome to benchmarking semantic search or semantic code retrieval on cloud code. My name is Kuba. I'm here from Turbo Puffer. For those unfamiliar with Turbo Puffer, we are a serverless full text and vector search database built from first principles on top of object storage. We serve some of the fastest growing AI companies in the world and if you'd like to know more about the talk or any question about Turbo Puffer or anything, just find feel free to find me after. So let's get started. So for those of you who are unaware, cloud code by default or I mean yeah, by default doesn't use semantic code search. There's actually a tweet from Boris. Those unfamiliar with Boris, he's actually like the founding father of cloud code and he talks about how cloud the early versions of cloud code did actually use semantic search with a local vector DB, but they just kind of found agentic search which is actually you know, grepping through your file system kind of worked better and you know, seems to work simpler for for cloud code. However, one of Turbo Puffer's customers actually fun fact, one of our very first customer is Cursor does use semantic code search and does index code bases into Turbo Puffer and you may think like this seems like a lot of work and it kind of is, but the reason they do this is because they see real performance gains because of this. They have this amazing blog post about their semantic search. They also have one about indexing code bases which I highly recommend as well. But you can see on the right the kind of performance gains they see with their composer model is like 24% increase in relative improvement in answer accuracy and I think it's like 12 and a half or 13% across all models and you know, I this benchmark came up before composer two, but you can probably imagine there's some sort of similar performance gains with that that new model as well. And in the bottom right you can see this is from to the same blog post. They talk about how in they perform an online AB test where they found that adding semantic either allowing or disallowing semantic code search led to like a 2.6% increase in code retention in large code bases and then 2.2% decrease in dissatisfied user requests. And you may also think like these numbers seem kind of small like what 2.6% 2.2%? Well, keep in mind that you know, out of 100 queries not all queries will ever really need semantic search or like benefit from it. So these they even state in the blog post that these numbers look small because not every you know, you can imagine a very simple tool call or very simple query would not really use semantic search. So what we think about here at Turbo Puffer and why we kind of really think vector search and why Cursor probably sees this like real performance gain is we think about how embeddings are cache compute and you can also you know, what is what do you mean cache compute? This sounds like a bunch of baloney you're throwing at me. You know, if we were like walk through like a cloud code trace and like a Cursor type trace of kind of the same thing, you can imagine that on the left side this kind of like a you know, grepping through the file system or what they call agentic search. How it just has like grep through this is for finding metadata filtering and understanding it in your code base. That's essentially grep through, read the files, grep through more if it doesn't find the right things and again, this is repeated on every session across you know, every agent running in for the same code base so that even if you're asking the same question or trying to do the same thing multiple times, you always have to do this compute again and you can see like I mean in this case 6,000 tokens isn't a lot in one time, but across every session and every agent this really starts to add up. And if we you know, on the right of this is more of a like like Cursor style or Cursor level trace where you have this upfront cost where you have to chunk embed and index the code base, but then you would essentially have this cache of the semantic meaning so that when a when a agent wants to understand you know, where or how does metadata filtering work, it can simply query you know, how is metadata filtered and get a lot get the chunks it needs and kind of the understanding it needs a lot faster and with token savings and again, this doesn't it's not a lot of savings in one time, but we all you know, we're not really running one agent anymore. Like I'm running like three at one time. So there's definitely some long-term savings. So what we did is we built a simple CLI tool for cloud code. We call it Turbo Grep. It's essentially you can imagine it's just a simple way of using a tree splitter library to essentially parse through your code base, chunk it, embed it using the Voyage code model and then upload it to Turbo Puffer. This is just a simple file system walk through. There is an open source library for the V1 version and then soon the V2 version will be open sourced. Here's a little video of example a tool call trace for a cloud code where you'll see it will call the T Puffer tool. Now it's called T Puffer search password reset token generator. This is in the Django repo. This is one of the repos I was testing it on and you can see that it gets the contents and then is able to kind of give the full explanation to the user. And you know, it's very easy to say like oh this works better, but the important thing is obviously benchmark it. Let's see for real what what how much better does it work? Cursor has their own internal context bench and there's this paper this public paper called context bench where essentially the benchmark is not really testing whether or not the coding agent solved or didn't solve a problem. It tests when it in the process of solving the problem, did it find certain files? Did it find certain lines and it find certain symbols? Because they you know, they kind of have this this thesis that like it you know, it's also important how you get there not just like the end goal. The process really matters for understanding like are agents actually looking for the right files? So essentially it's a human labeled data set of like in order to complete this task they the agent should have looked at this file, these few lines and these few symbols in order to like actually complete the task well. And I tested with three conditions. Essentially raw cloud code out of the box. Then I tested it with cloud code with a max of 50 line reads at a time and then the same thing with a windowed reads with the T Puffer search tool. You may ask me like why this like 50 line read thing limit? It's because it it became really noisy really fast if it's just reading like long files. It's really hard to like understand and get a difference because it all if it just reads a whole like a thousand line file, it doesn't really make a lot of difference in the numbers. So we'll start with the first result of it. This is precision. Precision is essentially the the measure of of how many files did it read like in its total process? How many of them were actually golden files? So if there was if if cloud read 10 files and eight of them were needed, it hits 80% precision. You can see the baseline it hit like 65% precision, 33% on line precision and 43% on symbol and it kind of goes up as we add windowed grep and then windowed grep plus semantic search got it to like 87% file precision for example. And this is also due to the fact like cloud code by default like it's a really exploratory. It loves to read as much as it can and like try to read everything for example and it kind of shows up in the 65% precision. And you know, it's hard to like translate these numbers, but a more like English or like human version of it is that you know, cloud code one in every three file reads is actually just a completely wasted file and with windowed grep it was one in five reads was a irrelevant file and then with semantic search there's only one in eight files was a you know, quote unquote wasted or irrelevant file and it kind of like scales up as you kind of add these tools. And you know, obviously this this is already like pretty good. Then we have recall. Recall is essentially how many of the needed files did it find? So for example, if there was 10 files in the task that it should have found, if cloud found five of them five files in its total trajectory and only three of them were actually these golden files, it would hit 30% recall. By default again, cloud code actually does win the file recall and again, part of this is because it just loves to explore every single file it can. Then we have line recall where it drops a lot more. This is kind of because it loved to read a lot of files, but also read to really read a lot of files that didn't have a lot of golden context lines. So you can kind of see that even though it did explore a lot, it also kind of explored the wrong things a lot and then with grep and grep plus semantic or windowed grep and windowed grep plus semantic search kind of have the same recall. You can kind of see on the right like kind of the what happened with the behavior between these three conditions. And you may be thinking like well, you know, semantic search didn't really like add improvements here. Like what what went wrong? Like actually some decreases. And we can like dig in a little further into the recall numbers for certain tasks. This is across 50 tasks by default, but if we break it down into where semantic search won you know, quote unquote won and where just like windowed grep won, we see some stark differences between like certain tasks performed a lot better with semantic search allowed and then certain tasks performed a lot better with with no semantic search and this kind of proves you know, certain tasks requires different types of tools. For example, when semantic search won, it was really good at finding a lot of behavior adjacent files that didn't have the same kind of keywords. For example, it was like I think one example off top of my head was we're trying to like it was a multi like first handle lots of different ORMs that have to like handle across different libraries and it didn't by default the keyword search didn't by default find all of them, but behaviors semantic search was able to kind of like understand these are all related files. And then grep won when it was really good at just the task was a lot of it like tracing through imports and if it like was able to find the keywords in like the first or second tool call, it was able to like just keyword search through that and find the relevant files. So again, it's like two different types of how to find files, but they kind of lead to like really different results. So in summary like what does this mean? Well, we like we saw semantic search like did boost precision quite a bit and we like kind of understand like grep and semantic search kind of find different code different ways. And an interesting thing to note as well was um these numbers weren't as great as Cursor's because part of like part of Claude code is it's built for just grepping. Like that's that's what Anthropic kind of focuses on. Like it's not built to under like to really understand like when to call semantic search or how to call it. Uh we kind of like add as an extra tool and it's like, "Hey, like here's this cool tool. You probably should use it sometimes." But it's very hard for it to have a true understanding of when to use it, why to use it uh versus like for example Cursor's composer they understand this is a built-in tool that um it knows when and how to use it and that's why they saw it's like 23 and 1/2 per uh percent uh performance gain. Uh so like in summary, uh you know, we think long-term winners were like kind of provide these lightweight tools to find the right context in various different ways. I think it's something important to think about. Uh you can't just like grep through everything unfortunately in a file system. Uh we think there's a lot of different ways to access lots of different types of information. Uh and the people that provide these like easy tools to provide to shrink down these billion context windows into the right million uh will win in the long term. Um that's the general talk. Thank you. If you have any questions, feel free to come up. \>\> \[applause\] \>\> I have my own idea of what semantic search is, but could you define what you what's your definition is of semantic search? \>\> Sure. Uh so this was just uh just doing vector search. It was just performing vector search using embedding it using Voyage's code model and then just embedding the the the query um query sentences or query tokens and just sending them back to to Turbo puffer. Yeah. Uh sorry, the uh embedding model Uh Voyage code three. Okay. And which vector DB? Turbo puffer. Yeah. \>\> Sorry. Uh no worries. Yeah, yeah. Yeah, for those for those who are unfamiliar with Turbo puffer, we are the we are the vector database that powers companies like Cursor, Anthropic, Notion. So when you use something like Cursor uh you have by proxy used Turbo puffer. Um so you may know us just not by name, I guess. I was just going to ask You may not have a benchmark for this, but how does it perform on code? Oh, that's that's tough. I mean it's hard to say \>\> better or is semantic better if the code's like a mess and you need to go and sort it out? Uh so I think it works best when there's a lot of like comments on code uh because it kind of finds that semantic meaning. \>\> like the documentation is kind of inline. Yeah, if it's like inline documentation, that was like a big um boost. I believe one of the repos I remember like looking through some directories and like asking Claude be like, "Why did it perform so well here versus not?" And it was one of it kind of explains me like um when I was looking through as well like those with like really good comments, for example, just like comments above the function, it's able to like really understand a lot more cuz you you kind of give this context to the model and the embedding model. So then it can like actually search better. Um cuz that's that's part of it. Like the the embedding it and is not the hard part. It's like figuring out what meaning really is of that chunk. Yeah. Yeah, you may have you you've mentioned some of it now, but of course semantic search is just similarity search. So if your query it doesn't really match the format of what you're querying against then you get some kind of innate distance. Do you do any kind of preprocessing on what your of the target data before you like do you do an parent-child where the parent is query-ish and the child is the real code or how do you \>\> In in this case it was just simple just just the code. Yeah, just the raw code just as a thing, but I can't speak for like how these more complicated and sophisticated customers use this, but I can imagine it's definitely something of providing not just code level meaning, but you know, as you said at least a parent-child relationship of like authentication flow, like that could be a good query to do a similarity search against, but the code itself is more like raw. Yeah. Yeah. Like Cursor has their own embedding model, which I think kind of helps with this of like how do you translate code into more of like a human level query. And I mean they're kind of been experts on that. I can't speak for how they do it, but I just know they do do it. I think they actually do what you said like they create fake comments on top of of your code and then embed the code with the comments. Yeah. So that's how they can have like their high recall when they So they add they kind of inject comments. Yes. Yeah, it's something that I think definitely could work. Yeah. In the back? Yeah. I was going to ask like how do you see uh I guess the vector database kind of working with the partners and how like when do you code like long-term or need to reduce the size I mean I think it depends. Like obviously the easiest way like people love grepping because it's zero cost. Like if you're able to like download everything to your local file system and just like grep through it, like yeah, that works. Um I think vector DBs are built for actually like multiplayer and like this like super maybe in a sense like hard to understand or uh complicated relationships between lots of data. For example, like a knowledge base like a Notion, like you can imagine kind of hard to like really grep through that really easily on your local machine. Like it's in a sense best to like have that vectorize uh for the agents. Um and even stuff like we have customers doing stuff like with multimodal data. You can't really grep through a video file. You can't grep through an audio file. You can't grep through an an image file. Like maybe you can glob on the on the file name, um but like get a true understanding of kind of multimodal data as well is that something that we find a lot of customers are doing. Um so it just kind of depends on the workload and um I I yeah, if you if you're hitting like at some sort of even like miniature scale like a vector DB kind of like helps offload a lot of this this work into like you know, cache computer cache this semantic meaning. Any other questions? Okay. Thank you all. \>\> \[applause\]