Raghav Saboo, Staff ML Engineer and Tech Lead for Search & Personalization at DoorDash, argues the main bottleneck in marketplace discovery is semantic understanding rather than engagement signals, illustrated by a ranker trained on engagement surfacing regular spaghetti for a gluten-free pasta query. As DoorDash expands into grocery, retail, pets, and gifting across billions of store-level items, it uses LLMs to interpret item meaning and shopper intent. His core pattern: run expensive LLM reasoning offline once, then distill it into small, fast serving models. He describes four building blocks: (1) LLM-generated relevance labels that improved retrieval NDCG by 2.3%; (2) semantic IDs — a learned taxonomy of the catalog that raised ranking MRR by 4–5% and powers query reformulation; (3) consumer memory at three timescales, stored as text, vectors, and graphs; and (4) steerable, LLM-generated personalized collections, which lifted order rate by nearly 1% in the pets category. ## Video Identity Observations ### Channel / Creator Self-Description (explicit claims) - The speaker self-identifies by role and employer: > "I'm Ragav. I'm a staff uh machine learning engineer at Door Dash. I work on search and personalization" - The content is a conference talk, not channel-native content: "Thank you everyone for coming to this talk and Devanch for organizing this." (Note: the organizer name is transcribed inconsistently and cannot be verified.) - The speaker claims first-hand production experience at DoorDash scale: "our catalog changes quite rapidly," "easily a few billion items at the store item level." - Crediting of collaborators suggests a team-effort framing: "thank you to all the collaborators at Door Dash who have helped ship a lot of these things." ### Claimed Scope (creator-stated) - The talk explicitly frames itself around **four primitives** for LLM integration: > "supervision, catalog semantics, semantic personalization, and steerable content generation" - Central thesis stated up front: "the real bottleneck is semantic understanding" for marketplace discovery, and "LLMs give us a genuinely new lever." - Concluding takeaways restated by the speaker: discovery is a semantic problem; "distill LM reasoning into primitives"; "shared representations create many use cases." - The title itself asserts the core architectural stance: **"Distill the LLM, Don't Serve It."** ### Intended Audience (evidence from presentation style) - **ML practitioners and engineers**, not general audiences — the talk assumes familiarity with: - Two-tower encoders, contrastive loss, curriculum training, hard negatives ("multi-level supervised contrastive loss") - Ranking concepts (NDCG, MRR, engagement towers, ordinal relevance towers) - Semantic IDs, embeddings, sparse ID features - **Industry/conference context**: references to prior talks ("As we heard previously as well," "We heard quite a bit about it in the previous talk") indicate a multi-talk technical event. - Business-leader-friendly metrics are included alongside technical depth (e.g., "close to 1% uh order rate increases and 6% in uh active users"), suggesting a mixed practitioner/decision-maker audience. ### Observable Presentation Patterns (demonstrated, not claimed) - **Pattern: distill, don't serve** is consistently demonstrated across all four sections — expensive LLM reasoning applied offline, then distilled into lightweight served models ("use expensive reasoning once offline then distill it into models that can you know be served cheaply and quickly"; fine-tuned "GPD 40 mini" labeler [as transcribed]). - **Concrete metrics used as evidence** throughout: +2.3% NDCG relevance, 4–5% MRR improvement, pets-vertical order-rate gains. - **Product-surface grounding**: examples repeatedly tied to real DoorDash features (Ask DoorDash, personalized collections, query reformulation for "Sriracha"). - No self-promotion of a personal channel, brand, or external content; the material is purely a technical conference presentation. ### Notes / Caveats - The transcript contains transcription artifacts ("Rexus" likely "rankers," "GPD 40 mini" likely a model name, "Gord Dash" likely "DoorDash"); these should not be treated as creator claims. - No curated metadata was supplied beyond the title. # Delivery Observation: Raghav Saboo — "Distill the LLM, Don't Serve It" (DoorDash) ## Overall Delivery Style The speaker delivers a **structured technical conference talk** in a conversational, explanatory register. The delivery is dense with system details, but he repeatedly anchors abstract concepts to **concrete shopping examples** before moving into model architecture. ## Tone & Register - **Measured and explanatory**, not performative. Technical claims are delivered evenly, with results stated matter-of-factly: > "we've improved NDCG of relevance by 2.3%." - **Inclusive framing** that references the audience's shared conference context: > "As we heard previously as well..." > "We heard quite a bit about it in the previous talk." - **Personal investment** surfaces occasionally through hedged enthusiasm: > "The second one that I'm particularly fond of is query reformulation." ## Recurring Vocabulary & Phrases - **"Primitive(s)"** — the structural backbone of the talk; introduced early and reused as section markers: "the four primitives," "our third primitive comes in memory," "the fourth primitive is steerable content generation." - **"Shared representations"** — a repeated thesis term: > "it's a system of shared representations that help us map this customer's journey." - **Distill / distilled / distilling** — appears across retrieval, ranking, and takeaways: > "distilling LM reason labels into our rankers." - **"You know"** — the dominant filler/softener, used extremely frequently throughout ("you know, I just adopted a puppy," "you know, our marketplace discovery"). - **Question-as-transition** — nearly every section is opened with a rhetorical question: > "how do we teach our retrieval and ranking systems what good means for different tasks?" > "What does the system know about the shopper and can that knowledge be reused across models again?" - **"Right?"** — used as a lightweight confirmation check after explanatory statements. ## Teaching Habits - **Claim-first structure**: the talk opens with an explicit thesis — "I'll start off uh today uh with a simple claim. Uh the claim is that... the real bottleneck is semantic understanding." - **Concrete consumer anecdotes before abstractions**: the puppy-adopting shopper and the gluten-free pasta search both precede their technical treatments. - **Running example reuse**: the hot-sauce example is invoked twice (taxonomy rigidity, then query reformulation — "Sriracha can lead to garlic chili garlic sauce or sambell Ole"). - **Numbered enumerations**: "four pieces," "three takeaways," "three time scales," "two sources of truth," "three places" — the talk is consistently scaffolded with counts. - **Contrast framing** to motivate solutions: "items that are merely related sit close to items that actually match," "human annotation... expensive slow and becomes quickly stale" vs. "behavioral signals. They're abundant but... biased." - **Closing with explicit takeaways**: "there are three takeaways I would love for you to take away from this talk," followed by a numbered summary and acknowledgments. ## Communication Patterns - **Hedged quantification**: metrics are frequently softened with qualifiers — "close to 1% order rate increases," "improving MRR between four to 5%," "a pretty high precision golden data set." - **Naming his own takeaways**: "This pattern has been pretty important for us over the past uh few years" (before stating the distill-once-serve-cheaply principle). - **Self-corrections mid-sentence** are common in delivery: "agentic inter interactions um oh sorry in session interactions," "we've also adapted uh or adopted um semantic ids." - **Credit-sharing at the close**: "thank you to all the collaborators at Door Dash who have helped ship a lot of these things." ## Uncertainty Notes - Repetition of "primitive," "you know," and question-transitions is strong enough to treat as stable habits. - Humor, gesture, and pacing cannot be assessed from transcript alone; the talk appears humor-free based on the text, but this is **low confidence** — laughter or audience reactions are not marked in the transcript. - Whether "fond of" moments reflect a broader enthusiasm pattern is a single instance; treat as isolated rather than stable. # Observation Record: "Distill the LLM, Don't Serve It" — Raghav Saboo, DoorDash ## Speaker Identification - **Speaker:** Raghav ("Ragav") Saboo, self-identified as a **staff machine learning engineer at DoorDash**, working on **search and personalization**. No other speakers are identified in the transcript; the talk concludes with thanks to unnamed DoorDash collaborators. --- ## Core Thesis - **Claim:** The real bottleneck for marketplace discovery is **semantic understanding** — knowing what items mean for users in context and what the shopper intends — not merely engagement optimization. - **Claim:** LLMs provide a "genuinely new lever," but should be used to produce **offline reasoning that is distilled into cheap, fast production models**, not served directly online. --- ## Context: DoorDash's Discovery Problem - DoorDash has expanded **well beyond restaurants** into grocery, retail, pets, gifting, and more, with the goal of capturing "every shoppable moment." - Users have **broad, multi-session shopping missions** (e.g., "I just adopted a puppy. What do I need to get started this week?"), unfolding across reformulations, collections, and multiple days. - **Stated difficulty:** capturing that arc "spans multiple models," motivating four primitives: **supervision, catalog semantics, semantic personalization (memory), and steerable content generation**. --- ## Primitive 1: Supervision (Graded Relevance) ### Stated Problem - Engagement-only ranking fails constraint queries: a search for *gluten-free pasta* might surface popular regular spaghetti or gluten-free white bread — popular or overlapping, but **not the right answer**. - **Qualification preserved:** two traditional sources of truth are presented as flawed — human annotation is "expensive, slow, and quickly stale"; behavioral signals are "biased by exposure, position, price, promotions and previous model choices." ### Stated Approach - Build a **high-quality seed set** of query–item pairs labeled on a 0–2 relevance scale by humans. - **Audit suspicious cases:** e.g., if a human labels an item irrelevant but it converts well, escalate to a **stronger LLM** with more granular prompts. - **Reconcile with other models** (query-to-taxonomy/category models) to adjust labels. - Fine-tune a **lightweight LLM** (cited example: "GPT 4o mini") on the golden dataset, then run it offline across the full catalog to generate graded query–item pairs serving as a **shared target for both retrieval and ranking**. ### Stated Guiding Pattern > "Use expensive reasoning once offline, then distill it into models that can be served cheaply and quickly." ### Reported Results (Retrieval) - **Two-stage contrastive method** on graded labels: - **Stage 1 (mining):** two-tower encoders with multi-level supervised contrastive loss for "global geometry shaping." - **Stage 2 (curriculum training):** mine hard negatives/strong positives the model misranks, relabel, retrain. - Claimed improvement: **NDCG of relevance +2.3%**, with downstream business metric gains. ### Reported Results (Ranking) - Added an **ordinal relevance tower** on LLM-graded labels alongside existing engagement towers (click, add-to-cart, conversion); shared bottom layers mean relevance is "distilled in the same backward pass." - A **value function** blends engagement vs. relevance per surface. - **Framing claim:** they are "not replacing the retrieval and ranking system with an LLM" but distilling reasoning into production architecture. --- ## Primitive 2: Catalog Semantics (Semantic IDs) ### Stated Problem - At "a few billion items at the store item level," **unique SKUs say nothing semantically**; the human-curated taxonomy is "too coarse and too rigid" (e.g., hot sauces taxonomy doesn't encode relationships between Huy Fong, Frank's, or Tabasco). ### Stated Approach - Adopt **semantic IDs**: learned hierarchical codes where **prefixes capture broad neighborhoods** and later tokens capture finer distinctions. Example given: hot sauces share prefixes but split into Mexican, Caribbean, Korean — learned **from data with zero labels**. ### Claimed Unlocked Capabilities 1. **Cross-category relationships:** chips, salsa, guacamole in different taxonomy branches can still cluster as one shopping mission. 2. **Cold start:** n-gram and byte-pair coding of tokens lets new items enter models as sparse ID features. 3. **Tail coverage:** sparse items inherit signal from semantically related items without waiting for exposure. 4. **Reverse audit:** human labels can be checked against learned semantic labels ("nice to have"). ### Reported Results - Ranker improvement: **MRR +4–5%**, translating to "pretty big conversion wins." - **Query reformulation** grounded in semantic neighborhoods (Sriracha → garlic chili sauce, sambal oelek); the speaker noted suggested queries without inventory are "meaningless," and the semantic-ID query graph drove "pretty massive" metric gains (metric type not specified in transcript). --- ## Primitive 3: Semantic Personalization (Memory) ### Stated Problem - User embeddings don't capture **why** a consumer has certain intents/preferences, and LLMs "cannot readily use these embeddings" — though he notes you could tokenize and train on them, an explicit "LLM-native counterpart" was found useful. ### Stated Approach - Consumer memory represented at **three time scales**: - **Long-term memory:** durable preferences from orders, searches, browsing, support interactions. - **Real-time context:** in-session interactions, cart state, active searches. - **Stated preferences:** explicit constraints from agentic interactions (e.g., "Ask DoorDash"). - **Memory blocks:** structured so new user dimensions (dietary, dining, substitution preferences) can be added, **decoupled** from downstream systems. - **Multi-materialization:** human-readable **text**, **latent embeddings** for retrieval/ranking, and **graph/tree (hierarchical)** structures linking consumers, brands, taxonomies. - **Context graphs** connect consumers to extracted memory concepts — described as fitting "sparse and multi-hop" consumer–item interactions, especially helpful at fine-grain taxonomy levels. ### Reported Results (Qualitative) - Graph-based embeddings claimed to **outperform existing taxonomy-based embeddings** for retrieval (no numbers given). - Three current applications: personalized collections, agentic personalization (Ask DoorDash), and encoded memory blocks as ranking/retrieval features. --- ## Primitive 4: Steerable Content Generation ### Stated Approach - Inputs (semantic IDs, memory blocks, LLM supervision) feed multiple model types; outputs include ranked semantic-ID lists, carousel titles, sub-copies across DoorDash surfaces. - **Personalized collections on store pages:** previously a fixed/attribute-based library; now an **offline, batch LLM process** generating title, subtitle, and items per consumer. - **Steerability claim:** collections react to occasions/moments (plant-based pantry rows, cat dry food for cat-only households, pantry staples during restocking sessions). - **Architectural note:** because generation is batch/offline, the **existing retrieval and ranking stack** still handles item hydration and collection ranking. ### Reported Results - Early tests in the **pets vertical**: "close to 1% order rate increases and 6% in active users" (described as early results). --- ## Concluding Takeaways (as stated by speaker) 1. **Discovery is a semantic understanding problem**, not only engagement; LLMs enable reasoning about item meaning and shopper intent. 2. **Distill LLM reasoning into primitives** — capture reasoning offline as labels, semantic IDs, and memory; let smaller, faster models serve it. His explicit claim: *"The online LLM call is often not the product architecture you need."* 3. **Shared representations create many use cases** — the same primitives power retrieval, ranking, and content generation. --- ## Unresolved / Unqualified Items in Transcript - Some metrics are given without baselines or measurement windows (e.g., "pretty massive MR gains" for query reformulation, metric abbreviated/unclear in transcript). - Pets-vertical results are explicitly framed as **early tests**, not settled outcomes. - Transcript contains transcription artifacts (e.g., "Rexus," "Gord Dash," "GPD 40 mini") — likely to "retrieval/ranking," "DoorDash," and "GPT 4o mini" respectively, but these remain as recorded. - Personalized collections are described as generated offline in one passage but with the ability to "build those collections as needed... at serving time" in another — both statements are preserved as expressed; the speaker did not explicitly reconcile this tension. # Video Attribution & Provenance Record ## Speaker Identification | Speaker | Role | Basis | |---|---|---| | **Raghav Saboo ("Ragav")** | Staff Machine Learning Engineer at DoorDash, working on search and personalization | Self-introduced in transcript; title in video metadata | | **Devanch** | Event organizer (acknowledged, not a speaker) | Credited in opening line: *"thank you... Devanch for organizing this"* | ### Speaker Structure - **Single-speaker conference talk.** No guests, interviewers, or other identified voices appear. - The speaker references a **previous talk** (*"As we heard previously as well"* / *"We heard quite a bit about it in the previous talk"*) — that presenter is **unnamed and not part of this video's attribution**. - Speaker credits **"collaborators at DoorDash"** in closing, unnamed. ## Provenance Boundaries ### Third-Party Claims & Tools Referenced - **GPT-4o mini** (transcribed as "GPD 40 mini") — cited as the fine-tuned lightweight labeler LLM. - **Techniques referenced without named sources:** two-tower encoders, multi-level supervised contrastive loss, byte-pair encoding (transcribed as "engram and bite parent coding"), semantic IDs. ### Company-Internal Claims (DoorDash — unverifiable from transcript) - NDCG of relevance improved **~2.3%** via two-stage retrieval training. - Ranker **MRR improved 4–5%** via semantic IDs, with "pretty big conversion wins." - Pets vertical personalization: **~1% order rate increase, 6% active user increase.** - Catalog scale described as "a few billion items at the store item level." - Tools/products mentioned: **Ask DoorDash**, personalized collections, query-to-taxonomy models. ### Uncertainty Flags - Metrics are self-reported by the speaker with no external verification, methodology detail, or links in the transcript. - Speech-to-text artifacts present (e.g., "GPD 40 mini," "Rexus," "Hoof" for Huy Fong, "syni syn synthesize"); correct names should be inferred cautiously. ## Promotional Content - **Flagged: promotional elements present.** The talk is a conference presentation by a DoorDash employee about DoorDash systems; it includes product mentions (**Ask DoorDash**) and self-reported business wins. No paid sponsorship or third-party product promotion is evident. ## Comments / Availability - No information about comments, Q&A, or platform availability is present in the transcript or metadata. **Recorded as unavailable.** Thank you everyone for coming to this talk and Devanch for organizing this. Uh I'm Ragav. I'm a staff uh machine learning engineer at Door Dash. I work on search and personalization and uh today I'm going to talk about uh the ways that we're integrating LLMs for you know our marketplace discovery uh specifically in four pieces and um how these primitives are kind of uh uh working towards our integration of LLMs into traditional uh Rexus as well. So I'll start off uh today uh with a simple claim. Uh the claim is that to build effective discovery in this case for a marketplace like Door Dash the real bottleneck is semantic understanding. So that is knowing what you know actually what items mean uh for users given some context and what a shopper actually intends to do on you know an app like Door Dash. And historically we have treated these uh as engagement optimization problems. Um however LLMs give us a genuinely new lever to extend this. So this talk is about uh you know the four primitives that have allowed us to start leveraging LLMs for problems at Door Dash. As you may know Door Dash is now grown uh well beyond restaurants. So we're in grocery, retail, pets, gifting and more. And our goal is to capture every shoppable moment. Uh and we know that across these verticals, the users frequently have very broad shopping missions, right? So for example, a consumer may say when they come to our app is, you know, I just adopted a puppy. What do I need to get started this week? And this consumer start may start on like searching for puppy food because they just adopted the puppy. That one query could unfold into further query reformulations. Um they land on collections for you know other shopping needs, complimentary items such as crates, leashes, uh training pads all the way to checkout. And this could happen over multiple sessions, mult multiple days, right? And capturing that whole arc is pretty hard because it spans multiple models. Um, and this is where the four primitives I will cover come in. Specifically, supervision, catalog semantics, semantic personalization, and steerable content generation. As we heard previously as well, it's a system of shared representations that help us map this customer's journey uh with the use of LLMs. Now, with supervision, the question we're asking or looking to address is how do we teach our retrieval and ranking systems what good means for different tasks? And here's a simple example, right? A shopper searches for gluten-free pasta. And if we only rank from engagement, very popular regular spaghettes might show up and because it sells well, you know, it may continue to show up. Even gluten-free white bread might be retrieved and, you know, show up because it overlaps with gluten-free. But neither of these are clearly the right answer. What we actually want is graded relevance. A true, you know, gluten-free pasta should be a high relevance item. Chickpea pasta might be a reasonable substitute and regular spaghetti while it's popular it fails the constraint that the user has. Now to solve this there are obviously two sources of truth. We could look at human annotation but that's expensive slow and becomes quickly stale because our catalog changes quite rapidly. Second is behavioral signals. They're abundant but obviously as I mentioned biased by exposure, position, price, promotions and you know previous model choices. So the missing signal is a scalable reasoning signal. This is where LLMs are very useful as they offer a way to produce that supervision for such tasks. So in our case, you know, the PIP pipeline starts with human ground truths. uh we build a high quality seed set of example query item pairs and on a three three uh level relevant scale 012. We audit the suspicious cases. For example, if a human label says you know that an item is irrelevant but that item performs very well on add to cart or conversions, we send that case to a stronger LLM uh to re-evaluate with more granular prompts. We also reconcile with other models. You know, for example, our query to taxonomy or category models. And if a query, you know, query maps to a set of valid categories and the items belong to those valid categories, we may adjust the label accordingly. And through this process, we're able to achieve a pretty high precision golden data set that we then fine-tune a lightweight LLM, let's say, for example, a GPD 40 mini on top of it. And that's when it, you know, really pays off. We are able to use that fine-tuned labeler offline across, you know, our full catalog to generate uh full graded query item pairs. And those labels become one shared target for both retrieval and ranking systems. This pattern has been pretty important for us over the past uh few years. um use you know expensive reasoning once offline then distill it into models that can you know be served cheaply and quickly. Now let's take retrieval first. The challenge is you know standard embeddings um at e-commerce scale quickly collapse the relevance uh distinction. items that are merely related sit close to items that actually match and the model uh may know two items are related but not be able to separate uh exact matches, substitutes, compliments. So our fix is a two-stage contrastive method um trained on uh the same graded labels. Stage one, which we call mining, does a global geometry shaping. We use two tower encoders uh with multi-level supervised contrastive loss. Um after that we use that base model to mine harder negatives and uh these are items that the model confuses hard negatives that are ranked too high or strong positives ranked too low. And the second stage after that is where we really put the model through curriculum training. uh use those hard negatives, relabel them and our uh you know relevance of the retrieval layer improves sharply after that. So after stage one we see overlaps between you know relevant and uh moderately relevant but after stage two there is a significant difference and this has been one of like our biggest uh levers in retrieval improvement uh across the board. um you know we've improved NDCG of relevance by 2.3%. And uh that's also been true with our downstream business metrics and we did the same with uh uh exercise with our ranking models as well distilling LM reason labels into our rankers. However, combining this with uh business and engagement objectives. Here we add a new tower uh in this case an ordinal uh relevance tower on top of the LLM graded labels and uh this prediction sits right alongside existing engagement towers uh click add to cart and conversion and since they share the same uh bottom layers the semantic fit and relevance fit gets distilled in the same backward pass and uh our model is able to predict uh you know probability of relevance across the different levels and we are able to blend uh through a value function on on top of this that allows us to uh fine-tune between engagement and relevance for different surfaces. Now these two examples are part of a larger theme of our work across similar projects uh where it seems right. We're not replacing the retrieval and ranking system with an LLM, but rather distilling the uh reasoning and understanding into some production ranking architecture. And this allows us to kind of quickly test and also scalably improve our models uh with LLMs while keeping uh you know the business and system objectives uh of these surfaces in mind. So once we have supervision, the next question is how do we represent a very large and constantly changing catalog uh in a way that every model be it language models or uh you know traditional retrieval and ranking models uh that they can understand and at you know Door Dash's scale uh you can imagine we're kind of surfacing items across stores across internationally actually uh now and you know that's easily a few billion items at the store item level and for these items you obviously have unique skew ids but it says nothing semantically. So we also have a human curated taxonomy that helps us classify and categorize these items into meaningful spaces. But often this is too coarse and too rigid. So for example something like hot sauces may just have like sauces hot sauces in the taxonomy uh but it doesn't tell you anything about the relationship between hoof or franks or tobasco. So that's where we've also adapted uh or adopted um semantic ids. Uh it's been introduced previously already. It's a backbone for a lot of our uh work uh now. And uh what it does is it gives us a you know a short hierarchical code that's analogous to our taxonomy but uh the ability to control the fine grain nature of it. And the result of this learn taxonomy is that each item gets this hierarchical code and the prefix captures broad neighborhoods and you know the later tokens capture finer distinctions. So for example in the map over on on the slide you'll see hot sauces and the structure emerging from the data with zero labels. each one share the same first and second prefix in this uh ID sequence but then split by specialtity. So now you've got Mexican, Caribbean, Korean hot sauces uh being split up. And the important property is that this code that's learned is you know comparable by as I said by prefix but also usable by all of our downstream models and that unlocks a few things for us. So the first is cross-category comparisons or relationships. You know chips, salsa, guacamole may live in different taxonomy branches but a shared semantic ID neighborhood can uh indicate that they belong together and maybe a shopping mission. Second cold start problems. uh oftentimes we get new catalog items added by stores and with techniques like engram and bite parent coding of these tokens we're able to scalably add these to our models as sparse ID features third tail coverage so sparse items uh and uh inherit a lot of signal from semantically related items and uh you know we don't have to wait for the volume or exposure uh to you um for to consumers for these items. And then lastly, nice to have is the fact that we can actually also do a reverse audit. So we can audit our catalog and how well our human labels agree with the semantically learned labels. So a couple of examples of where we're using it today uh just to get give some real metrics. Um it has been one of our biggest improvements to our ranker. Um in in this case it's actually improved our uh ranker by uh improving MRR between four to 5% and that's translated to pretty big conversion uh wins as well. The second one that I'm particularly fond of is query reformulation. So again going back to the hot sauce example, Sriracha can lead to garlic chili garlic sauce or sambell Ole and because these queries map onto like our catalog grounded semantic neighborhood uh that results in much more relevant queries that we're suggesting to users. Um ultimately if these are queries that we don't have inventory for on Door Dash, they're meaningless. And with the addition of semantic ids in this query graph, we were actually able to see pretty massive um MR gains as well uh for for this piece of work. So for now we've talked about you know item mapping and relevance and the next question is consumer context. What does the system know about the shopper and can that knowledge be reused across models again? And this is where uh our third primitive comes in memory. So memory in the agent context is pretty well understood now. Um we apply the same concepts to uh our recommendation systems. So user embeddings are clearly very useful. uh we have many representations of our consumers but they don't necessarily get to uh why a consumer may have certain intents um and why they might have certain preferences. Additionally, LLMs cannot uh readily use these uh embeddings. You could tokenize and like uh train these models to learn it. But oftentimes having some explicit LM native counterpart is very useful and that's what we found. So the idea of memory is to represent uh the consumer in multiple forms semantic in you know inspectable and reusable and uh the way we think about memory is uh in three time scales. So long-term memory captures durable preferences from orders, searches, browsing, support interactions. Real-time context captures uh agentic inter interactions um oh sorry in session interactions so cart state active searches and then we have stated preferences which come from agentic interactions so something like ask door dash for example where consumers are able to explicitly state their constraints and preferences as well and the way we represent consumers in uh in uh this long-term memory is through uh memory blocks. And these are structured in a way that allows uh us to add new dimensions of the user, you know, from dietary preferences to dining preferences to uh substitute pre uh preferences um as we learn more about the consumer and that is uh decoupled from like the downstream system that u that doesn't need to reinterpret uh this for their own use cases. So each consumer's memory is multi materialized in multiple forms. So first is text that's human readable. It's uh captures the consumer in a compact way. Second is are latent vectors embeddings of those memory blocks that can be fed into retrieval and ranking systems. And third are graph and tree or hierarchical approaches. So uh you know type relationships between consumers and brands taxonomies and memory uh revealed um preferences and these are assembled into you know composite memory objects that both ML models and LLM use downstream. So an example is you know this context graph uh where we connect consumers with extracted memory concepts and a graph needs fits neatly into shopping journeys because you know oftent times uh consumer item interactions are sparse and multihop. And in this case these context graphs are able to link consumers across memory concepts that we previously did not have uh relationships for. And it's particularly helpful uh at fine grain taxonomy levels. In our case, we're seeing uh for retrieval uh uh using uh graph uh based embeddings outperforming our uh existing taxonomy based embeddings. And today this memory framework shows up in three places. The first is personalized collections. I'll talk about that a bit later. Uh second is agentic personalization. So ask door dash for example uses this to personalize its sessions uh for you. And third in uh retrieval and ranking models as I said we encode these memory blocks and feed them as features in in downstream models as well. So the fourth primitive is steerable content generation. We heard quite a bit about it in the previous talk and this slide is my take on how we're p piecing all these uh primitives together. So all of these inputs, semantic ids, memory blocks um and graded relevance or um LLM supervision feed into multiple models you know be it LLMs or small language models or um uh you know traditional models and from these inputs we can generate different output shapes you know ranked semantic ID lists, carousel titles, subcopies and these show up across different surfaces today on Gord Dash and A direct example of this is uh our um personalized collections on store pages. So historically collections on store pages uh on Door Dash have been a fixed library or attribute-based um and we've been able to kind of expand that through uh consumer level collection generation. And today this is an offline uh LLM process where we take in consumer memory semantic ids and syni syn synthesize uh collections um all the way from title subtitle to the actual items. And the important part is that control and steerability. We're able to react to occasions and moments um and build those collections as needed uh for different consumers at serving time. uh because this is all batch and uh generated offline through LLMs, we are able to still use our uh existing retrieval and ranking stack for item hydration, collection ranking, etc. And you know this is for example what a shopper actually sees, right? For a shopper with plant-based affinity, the system can generate plant-based pantry rows. For cat-only households, it can generate cat dry food rows. And if a session indicates that you know a person is going through pantry restocking it can bias towards pantry staples. And our early tests show that you know consumers are feeling the benefits of these tailored experiences. Uh an example is uh within our pets vertical we've been able to drive uh close to 1% uh order rate increases and 6% in uh active users. So to conclude uh there are three takeaways I would love for you to take away from this talk and how LLM fit into search and recommendations. First discovery is a semantic understanding problem. It's not only about engagement. LMS give us a way to reason about item meaning and shopper intent. Uh in in Door Dash's case for example. Second uh distill LM reasoning into primitives. capture you know reasoning uh offline as labels, semantic ids, memory and then let smaller and faster models serve it. The online LLM call is often not the product architecture you need. Third, shared representations create many use cases. Uh once you have these primitives, they can power, you know, retrieval, ranking, content generation, etc. So last but not least, thank you and thank you to all the collaborators at Door Dash who have helped ship a lot of these things as well.