Talk by Hanna Lichtenberg (AI Engineer at Mixedbread AI) on improving retrieval for agents. It examines why agents often perform poorly with semantic search, arguing that models default to writing queries suited to weak, familiar lexical tools (e.g., grep or BM25) and hit those tools' limits, even though knowledge tasks like legal and financial research require better retrieval. The session covers co-designing agents with state-of-the-art retrieval tools so they can effectively use stronger search beyond simple keyword matching. # Channel Presentation Observations ## Video Identity - **Title:** "How we taught agents to use good retrieval - Hanna Lichtenberg, Mixedbread AI" - **Format:** Dual-speaker technical presentation (Hanna, Amir) — appears to be a conference or recorded company talk. ## Explicit Self-Description (Creator Claims) - **Speaker affiliations:** "I'm Hannah. I'm an AI engineer at Mix bread and I'm leading the agentic search at Mix bread." / "I'm Amir. I'm the co-founder of Mix bread." - **Internal framing:** The team calls the reasoning-vs-retrieval gap "closing the Oracle gap with knowledge agents" — "Internally, we call this gap... the knowledge gap." - **Product claims:** Their beta search agent is "top one on the snowflakes match QA benchmark, achieving an accuracy of 93.4" — a verifiable, specific claim. - **Recruiting signal (explicit):** "if you want to be part of pushing the boundaries of retrieval even further, we are happy awaiting your application." ## Intended Audience - **AI/ML engineers and researchers:** heavy use of technical vocabulary without explanation (NDCG, BM25, on-policy RL, supervised fine-tuning, rag/semantic search, benchmarks like BEIR and BrowseComp). - **Job-seeking technical talent:** the closing line directly addresses potential applicants. ## Claimed Scope - **Domain:** Agentic retrieval / search tooling for LLMs — "the retrieval tools are the main access pattern for this reasoning." - **Beyond code search:** explicitly aiming "to work beyond code search for knowledge work" (legal, finance). - **Product positioning:** built on their own platform — "our harness is fully built on a mixed back platform" — with a beta in production. ## Demonstrated Presentation Patterns - **Benchmark-driven evidence style:** claims supported by named benchmarks (BrowseComp Plus, OfficeQA Pro, Snowflake Mass QA) with concrete numbers (e.g., GPT-5 with Mixedbread "just three point" from Oracle on BrowseComp). - **Transparency about limitations:** "Our trained agent is not released yet, unfortunately. However, we have some intermediate results." - **Attribution of external work:** credits benchmarks to originators — "Office QA Pro was created by Databricks and BrowseComp was created by OpenAI." - **Pedagogical structure:** problem framing (knowledge gap) → diagnosis (bad agent queries, e.g., "senator woman questions billionaires...") → solution (harness design, training with retrieval + trajectory rewards). ## Notable Gaps - No channel-wide description supplied (no curated metadata present); observations limited to this single talk. - **Unreleased claims:** headline results (NDCG@10 of 0.4 vs. 0.18 for the GPT multi-hop agent) are self-reported and pre-release. # Delivery Observations: "How we taught agents to use good retrieval" — Hanna Lichtenberg & Amir, Mixedbread AI ## Format & Structure - **Two-speaker tag-team delivery**: Hanna opens ("Hey, today we're going to talk about... I'm Hannah"), Amir takes the problem/motivation section, then Hanna returns for the harness and training sections. The handoff is marked in the transcript: ">> This motivated us to make our own search agent..." - **Lecture-style signposting**: explicit enumerated structure, e.g., "there are five more main points for this", "The first step for building this agent was to define a very powerful harness", "The second important step into building our own search agent was of course the training itself." ## Tone & Register - **Casual, conversational opener**: "Hey, today we're going to talk about how we taught agents to use good retrieval or how we call it internally closing the Oracle gap." - **Direct, punchy conclusions**: "Meaning that actually the bottleneck here is not the reasoning. It's actually the access to the right knowledge." ## Vocabulary & Framing Habits - **Internal/proprietary naming**: repeatedly coins terms and attributes them to internal usage — "how we call it internally closing the Oracle gap", "Internally, we call this gap... the knowledge gap." - **Benchmark-heavy vocabulary**: BEIR/BEIR Nano-BEIR, BrowseComp/BrowseComp Plus, Finance QA Pro ("Office QA Pro" in the transcript, likely a transcription artifact), Oblique Congress benchmark, Snowflakes Mass QA — used as primary evidence throughout. - **Numeric specificity**: "93%", "64%", "nine points", "eight points", "NDCG of 10 at 10 of 0.4", "accuracy of 93.4". ## Recurring Phrases & Patterns - **"So," as a frequent connector**: "So, we have seen over the past years...", "So, there's a huge gap...", "So, the model tries to just write...", "So we see this knowledge gap in real world." Occurs dozens of times. - **Repeated "right"/"yeah?" check-ins**: "just think back how good GPT-3.5 was and how good GPT-5.5 was, right?", "grep just tries to find regular expressions, right?" - **"basically" as a filler/simplifier**: "if you look at search over the last 20 years basically", "It's like real world deep search task basically", "the treasuries of the US treasuries of the past um 100 years basically." - **"um" fillers cluster densely**, especially in Hanna's later sections: "Um it has a filter chunks tool...", "Um and yeah, the next search one can start." ## Teaching Habits - **Explain-by-example before generalizing**: shows a concrete failed query first — "senator woman questions billionaires not a company then okay thank you staff will check hearing. It's basically gibberish, right?" — then explains the three causes. - **Enumerated causal lists**: "The second thing is the models are trained to use the web pretty efficiently... And number three, obviously, is the benchmark bias." - **Definitional teaching**: pauses to define terms inline, e.g., "here's a Oracle performance in dotted lines and Oracle means what is the maximum theoretical performance of the models..." - **Contrast framing**: repeatedly juxtaposes exponential LLM growth vs. slow search improvement, and Oracle performance vs. noisy-corpus performance. ## Uncertainty Notes - Speaker attribution beyond Hanna's opening and Amir's self-introduction is **not always clear** from the transcript; the mid-video handoff ("This motivated us...") suggests, but does not confirm, who delivers each section. - Transcript artifacts (e.g., "GPT-5.5", "Mix bread", "Office QA Pro", "a scrap" for "grep") limit confidence in exact wording; quotes above are taken verbatim from the transcript as supplied. - The single ">>" turn markers and non-verbal notes ("[clears throat]", "[snorts]") indicate live spoken delivery with minor disfluencies, but not enough repetition to call these a pattern. # Position Extraction: "How we taught agents to use good retrieval" ## Speakers - **Hanna Lichtenberg** — AI engineer at Mixedbread, leading agentic search - **Amir** — Co-founder of Mixedbread - Transcript does not consistently mark speaker boundaries; attributions below reflect the flow of the talk. ## Central Thesis: The "Knowledge Gap" - **Reasoning is growing exponentially** while **search improves very slowly**. Speaker cites the trajectory from GPT-3.5 to newer models ("GPT-5.5" as stated) versus 20 years of slow retrieval progress. - Retrieval tools are the **main access pattern** for LLM reasoning in knowledge work (legal, finance). - Internally at Mixedbread this divergence is called the **knowledge gap** — and the speakers claim it is validated by benchmarks, not just theory. ## Benchmark Evidence | Benchmark | Oracle | Codex (default tools) | Gap closed via Mixedbread | |---|---|---|---| | **BrowseComp Plus** (OpenAI's BrowseComp with fixed 100k-doc corpus) | 93% | 9% | Within 3 points of Oracle (GPT-5 + Mixedbread) | | **OfficeQA Pro** (Databricks; US treasury docs, ~100 years) | 64% | 8% | Gap "almost completely closed" | - **Claimed conclusion:** the bottleneck is not reasoning but **access to the right knowledge**. - Caveat: figures are as stated by the speakers; "Oracle" is defined as maximum theoretical performance given the right documents. ## Why Agents Write Bad Queries Speaker presents a real example of gibberish keyword-stacking: *"senator woman questions billionaires not a company then okay thank you staff will check hearing."* Three stated causes: 1. **Training for coding tasks** — models optimized for grep-style regex exploration. 2. **Mimicking human web-search behavior** — keyword-on-keyword patterns. 3. **Benchmark bias** — BEIR / Nano-BEIR use "Caveman-style" entity queries that structurally favor **BM25**, so agents guess keywords to maximize overlap. ## Mixedbread's Search Agent Design **Goals:** precise, fast, cheap; work beyond code search; use the right tool per use case. ### Harness (built on Mixedbread platform) — four tools: - **Overview search** — wide semantic search, up to 50 chunk *summaries* - **Semantic search** — full payload of top 10 chunks - **Filter chunks** — metadata-facet sorting - **Grep** — exact keyword matching ### Agent loop: - Max **four search rounds**, with **parallel searches** per round - Initial context: user query + initial semantic results + metadata facet hints - Query split into up to four aspect-covering queries; tool chosen per intent - Deduplicated chunks; outputs ranked relevant chunks ### Five stated reasons the harness yields better queries: 1. **Goal framing** — agent articulates needed evidence before querying 2. **Distinct tools** — semantic vs. exact-match selection 3. **Task framing trick** — instructing "write one concise sentence describing what you want to find" instead of "write a search query" 4. **In-prompt examples** of good queries and aspect-splitting 5. **Preview of corpus** via original-query semantic results ## Training Approach - **Small LLM** for speed; optimizing tool choice, semantic query quality, exploration, ranking, efficiency. - **Pipeline:** supervised fine-tuning from a larger teacher LLM → **on-policy RL** with a custom search reward. - **Retrieval reward:** NDCG of final ranked list + LLM-judge rubrics (relevance, all-chunks relevance, plausible ranking). - **Trajectory reward:** LLM-judge rubrics (natural-sentence queries, sufficient exploration — not too much or too little). ## Results (as claimed) - **Oblique Congress benchmark:** NDCG@10 of **0.4** vs. **0.18** for the best paper model (GPT multi-hop agent). - **Production beta** ("Mixedbread agentic search"): top-1 on Snowflake's **Mass QA / match QA leaderboard**, **93.4 accuracy** with Gemini 3.5 Flash — achieved "with way less effort than comparable LLMs with other search agents." ## Qualifications & Unresolved Points - **The trained agent is not yet released** — results described as intermediate. - Benchmark provenance is stated as: BrowseComp (OpenAI), OfficeQA Pro (Databricks); the leaders attribute the gap finding to "real benchmarks and real tasks." - Stated closing message: results show "a lot of room for improvement" for large models and their search tools; ends with a recruiting note ("happy awaiting your application"). # Speaker Attribution Record ## Video **Title:** How we taught agents to use good retrieval - Hanna Lichtenberg, Mixedbread AI **Video ID:** `01a15ca8-d27e-7f6c-9eb5-83d7c1b3b35a` ## Speakers | Speaker | Role | Evidence | |---|---|---| | **Hanna Lichtenberg** | **Creator / Presenter** — AI engineer at Mixedbread, leading agentic search | *"I'm Hannah. I'm an AI engineer at Mix bread and I'm leading the agentic search."* | | **Amir** (surname not given) | **Co-presenter / Guest role** — Co-founder of Mixedbread | *"I'm Amir. I'm the co-founder of Mix bread and let's jump into it."* | > Note: Speaker turns are not marked in the transcript; attribution of specific passages to Hanna vs. Amir beyond the self-introductions is uncertain. Most of the body content uses "we/we". ## Referenced Sources & Third-Party Claims - **Benchmarks:** BrowseComp (created by OpenAI), BrowseComp Plus (fixed-corpus variant), OfficeQA Pro (created by Databricks), BEIR / NanoBEIR, "Oblique Congress" benchmark, Snowflakes MTEB MatchQA leaderboard (name as spoken; transcript unreliable on exact naming) - **Models/tools referenced:** GPT-3.5, "GPT-5.5", Codex default tools, GPT-5 with Mixedbread, GPT multi-hop agent, Gemini 3.5 Flash, BM25, grep - **Claimed results (speaker-reported, unverified):** Oracle 93% (BrowseComp), 64% (OfficeQA Pro); Codex default tools 9 and 8 points; NDCG@10 of 0.4 vs 0.18; 93.4 accuracy on MatchQA ## Promotional Segments - Discussion of **Mixedbread platform, harness, and search agent** throughout is product-promotional in nature - Closing line: *"we are happy awaiting your application"* — explicit recruitment/promotion ## Availability & Uncertainty - **Trained agent not released:** *"Our trained agent is not released yet, unfortunately."* A beta version ("mixedbread genetic search") is stated to be in production. - No curated metadata supplied; no comment availability information. - Exact spellings of names, benchmarks, and metrics are uncertain from the transcript alone. Hey, today we're going to talk about how we taught agents to use good retrieval or how we call it internally closing the Oracle gap with knowledge agents. \>\> I'm Hannah. I'm an AI engineer at Mix bread and I'm leading the agentic search at Mix bread. \>\> Yeah, I'm Amir. I'm the co-founder of Mix bread and let's jump into it. So, we have seen over the past years with the LLMs just getting better and better that the reasoning capabilities of the models is growing exponentially. So, just think back how good GPT-3.5 was and how good GPT-5.5 was, right? So, I can clearly exponential curve. But, if you look at search over the last 20 years basically, we see that it's improving, but it's improving very very slowly. So, there's a huge gap between how basically LLMs and the reasoning is evolving and how retrieval is evolving. But, the retrieval tools are the main access pattern for this reasoning there to get the right knowledge and to be truly useful beyond code like in legal work, in finance work and so on and so forth. Internally, we call this gap happening right now between reasoning and search the knowledge gap. And it's not just one obscure theory of ours. We see that actually with real benchmarks and real tasks that this gap exists in real world as well. Let's take two benchmarks here, BroSque Plus and Office QA Pro. BroSque Plus is like a browsing benchmark where we have pre-complex queries and a corpus of 100,000 documents and we just try to answer these questions. It's like real world deep search task basically and Office QA Pro which has the treasuries of the US treasuries of the past um 100 years basically and we ask really complex question over this. Office QA Pro was created by Databricks and BrowseComp was created by OpenAI and BrowseComp Plus is a version of it with a fixed size corpus while BrowseComp was over the open web. And we see here's a Oracle performance in dotted lines and Oracle means what is the maximum theoretical performance of the models if you would put in the right documents with the question with with the question. We see for Office for BrowseComp it's 93% and for Office QA Pro it's 64%. And now we take something like Codex with its default tools and want to see hey, what is the performance this tools get right now? And you see there's a sharp drop in the quality of the answers Codex produces. For BrowseComp it's nine points and for Office QA Pro it's eight points. So we see that the models are extremely capable if they would get the right documents but if you put them into the noisy corpus the performance drops sharply. Meaning that actually the bottleneck here is not the reasoning. It's actually the access to the right knowledge it needs to answer this question. So we see this knowledge gap in real world. And if you would just drop in way better search using Mixbread with just a search tool using latent interaction we can recover most of the performance. So for BrowseComp the difference between the Oracle and the uh GPT-5 with Mixbread is just three point and for Office QA Pro we even almost completely closed the gap. With giving the model better search tool we can recover most of the knowledge gap. But looking at the queries the model asks or writes right now to the search system, it gets super super interesting. So, here's an example query we found uh during some benchmarking, which is senator woman questions billionaires not a company then okay thank you staff will check hearing. It's basically gibberish, right? If you have a search system which wants to have neural questions or semantic question and just gets these keywords over keywords, it gets confused. And the reason why the models write these type of queries is they're mostly trained for coding task, like for coding agents, which are then optimized for code base exploration using tools like grep. And grep just tries to find regular expressions, right? So, the model tries to just write as many expression it thinks are in the document to find the right thing. The second thing is the models are trained to use the web pretty efficiently. And to use the web tools properly, which are highly optimized for humans, they try to mimic human-like query patterns. So, just keyword on keyword on keyword. And number three, obviously, is the benchmark bias. Most benchmarks we have right now, like Beer, Nano Beer, use Caveman's style queries, which are entity-based queries that structurally favor heavily BM25. So, right now, the agent guesses the keywords to actually increase the overlap between the query and documents and can't really use powerful search tools properly. \>\> This motivated us to make our own search agent, to teach it to use powerful search properly, especially to use the right search tools for different use cases. Um and to work beyond code search for knowledge work. And most importantly, the agent should be precise, fast, and cheap. The first step for building this agent was to define a very powerful harness. Um \[clears throat\] our harness is fully built on a mixed back platform. Our agent has four main search tools, which are overview search. This is used as a very wide semantic search where the agent receives up to 50 retrieve chunks. Um and it sees only summaries of the chunk contents to really have just like an overview of the corpus every uh what exists and to not fill up the context too much. Then there is the main semantic search tool where the agent gets the full payload of the top 10 retrieve chunks. Um it has a filter chunks tool where it can sort sort and find chunks based on metadata facets. And of course we also have a grep tool for the keyword match searches. Our um agent loop, the harness itself is very simple and short cuz the agent should be fast. But we still wanted it to uh have a lot of exploration possibilities. That's why we decided to um define that it has a maximum four search rounds, but within each search round it can have parallel searches. So it can start several search at that searches at once. Um initially our agent sees the user query, but also already um results from an initial semantic search on the user query plus um hints on which metadata facets are available. Um so that based on this preview of the corpus the agent can start its search planning. It splits its search intent into maximum four queries um which cover separate aspects. And then for each query and each search intent, it will pick the best tool. Um as you can see like an example here on the right. Then when um the results of the tools are returned to the agent via deduplicating chunks, so that it's never seeing several chunks um several times to not fill up the context. \>\> \[snorts\] \>\> Um and yeah, the next search one can start, and as soon as the agent has enough evidence to answer the user query, it submits its ranking. Um yeah, we are just outputs all the chunks that are relevant to the query in a plausible ranking. So, why does this harness encourages better semantics queries and like better use of search tools? Um there are five more main points for this, which is first the goal framing. The agent has to articulate articulate what evidence it needs before writing the query. Um then the different tools are very important, so it can really use semantic search only if it needs aspects, and also only use a scrap when it needs exact keyword matching. Um then the way in which we frame the task of query uh writing is an important point. We kind of trick the um model into not thinking it has to write the typical BM25 base query by just instructing it to write one concise sentence describing what it wants to find, instead of directly instructing write a search query, so it cannot fall into this old pattern. Um then we put also provide examples in a prompt just to show the a few good queries and also how to divide an input query into different aspects to explore um the corpus. Yeah. What also helps is that we provide the original original query semantic search results. So, it can see what the corpus is about and really define um see the language, define where to dig deeper in which aspect. The second important step into building our own search agent was of course the training itself. So, here for training, we just decided to have a very small LLM to make the agent even faster. So, we are training the small agent um to optimize the search strategy itself to have better tool choice, um higher quality of semantic queries, more exploration ranking, and to be more efficient. So, the first step in training is um supervised fine-tuning with a larger teacher LLM. And then we uh did on-policy reinforcement learning with an own search reward. Our search reward is a combination of both of a retrieval reward and a trajectory reward. The retrieval reward um is based on like the metric result, typical retrieval metric results like we call it NDCG, that's the agent's final ranked list achieves. Plus also um an LLM retrieval judging. We had thought like it gets several rubrics where it decides um if the rubrics is hit or not, the rubric is hit or not, which are if the agent result relevant for this query, are all chunks relevant, and is the ranking itself plausible. For the trajectory reward, we also using an LLM judge and here um is the point to like really improve the tool queries, make it more efficient and to um improve the quality of the queries. Of course, like we have rubrics that are where the judge is deciding if the query is really a natural sentence. Also, if the amount of exploration is sufficient, is it too much or too less. Here you can see an example trajectory. You see that in the beginning we have the initial search and the initial metadata hints. Um then the agent did four parallel searches in the first round and then a second round just a simple grep tool. And here's an extract of um a trajectory. Here we see the agent queries. Um the input query is on the left. This is a very long query typical rambling style query from the Oblique Congress benchmark. And yeah, we see the agent queries here for the first semantic search tool. It's really a sentence describing what it wants to find and not a weird keywordy added behind each other formulation. The grep tools of course are like typical keyword patterns, which is exactly as intended. Our trained agent is not released yet, unfortunately. However, we have some intermediate results. Here on the left side for the Oblique Congress benchmark, we see that we achieved an NDCG of 10 at 10 of 0.4, which is a huge jump um towards the model that performed best on the paper of this benchmark, which is the GPT multi-hop agent, and which achieves 0. 18. On the right side, you can see the result of our beta version of the search agent, which we have right now in production. It's the mixed better genetic search. And this agent is top one on the snowflakes match QA benchmark, achieving an accuracy of 93.4 when we give the Gemini 3.5 flash model our genetic search as search tool. And this uh this performance was also achieved while having way less effort than comparable LLMs with other search agents. You can check out the match QA leaderboard. These results show that there's still a lot of room for improvement when it comes to huge language models and their search tools. And if you want to be part of pushing the boundaries of retrieval even further, um we are happy awaiting your application.