Spotify talk on adapting LLMs for music and audio personalization. Yves Raimond (SVP, GM of AI & Personalization) covers Spotify's shift to generative personalization — a steerable DJ, prompted playlists, editable taste profiles, and personal podcasts — built around the Large Taste Model used daily by one in four US Premium subscribers. Jacqueline Wood (Staff ML Engineer) explains NEO, Spotify's four-stage training recipe: semantic IDs added to an open-weight LLM (Qwen) with a frozen backbone, multitask instruction tuning (including cold-start audiobook recommendations), decoding behavior, and grounding LLM judges with user profiles to improve agreement with human ratings on ambiguous queries. # Video Identity Observations ## Source - **Title:** *Teaching LLMs to Speak Spotify — Yves Raimond & Jacqueline Wood, Spotify* - **Format evidence:** Conference-style talk with two named speakers from Spotify, references to a moderator ("thank you Deven for inviting us"), a "previous talk" on semantic IDs, and closing remarks referencing "all the talks today" — indicating a multi-talk industry/ML conference setting. ## Intended Audience - **ML/AI engineering and industry practitioner audience**, evidenced by deep technical content without introductory framing: - Discussion of fine-tuning stages, LoRA, constrained decoding, beam search, top-p sampling, and ablation studies. - Reference to academic work assumed known: *"generative semantic ID retrieval, tool-based LLM recommenders, the plum paper"*. - Cross-talk continuity: *"semantic IDs were presented in the previous talk"*. - **Practitioners building recommender systems**, reinforced by the closing framing: *"if you want to power language steerable personalized recommendations for your user, this is how we taught openweight LLMs to speak Spotify."* ## Claimed Scope - **Topic scope is explicitly stated:** *"making LLMs speak Spotify and how we turned our recommendation system on its head to be LLM native."* - **Two-part structure announced by the speakers:** - Yves: history, product vision, and the shift to *"generative personalization"*. - Jackie: *"how these models are actually trained"*, including the **NEO** four-stage training paradigm, ablations, inference trade-offs, and LLM-judge evaluation methodology. - The talk claims both **product-level** scope (Spotify DJ, prompted playlists, taste profile, personal podcast) and **systems/research** scope (training recipe, evaluation, production deployment). ## Self-Description Evidence (Creator Claims) - Speaker credentials stated on-camera: > *"I joined Spotify a year ago. Before that I was at Google working on personalization for Google search and prior to that I was working on personalization at Netflix."* > *"I'm Jackie or Jacqueline, a staff machine learning engineer at Spotify."* - Scale and production claims made by the creator: - ~*"760 million active users monthly in about 184 markets"* and *"a bit more than 100 million music tracks"*. - *"one in four US premium subscribers interact with that system on a daily basis."* - *"NEO is actually the first example of combining all these capabilities into one system… and we are using this in production today."* - Evaluation claims: 75% LLM-judge/human alignment for podcast recommendations, 91% alignment improvement on ambiguous queries, and 0.87 agreement for Cranfield-style collections. ## Demonstrated (Observable) Patterns vs. Claims - **Demonstrated:** The material itself shows a technical conference presentation structure — problem framing, product examples, training methodology, ablation results, and evaluation discussion. - **Claims requiring external verification:** all usage statistics, performance gains ("huge wins online"), production status, and "first" framing are creator assertions, not independently verifiable within the video. ## Not Present - No curated metadata was supplied beyond title and transcript. - No subscriber-oriented channel branding, calls to action, or channel self-description (e.g., "about this channel") appears — this is a single conference talk, not a channel-branded upload. # Delivery Observation: "Teaching LLMs to Speak Spotify" ## Overall Tone - **Conversational-professional conference register.** Both speakers use informal fillers ("um," "uh," "so") but anchor the talk with formal signposting, typical of a technical conference presentation. - **Enthusiastic but measured when discussing results.** Excitement appears mostly around launches and adoption metrics: - *"that's pretty exciting"* - *"another one that I'm really excited about"* - *"we saw huge wins online with this."* ## Recurring Phrases and Vocabulary - **"Generative personalization"** — the central coined term, introduced, defined, and reused throughout: > *"the phase that we entering now... is something that we call generative personalization"* - **"Matching problem"** — repeated as the framing device for the recommendation challenge: > *"So the matching problem uh is uh actually surprisingly uh complicated."* - **Steering / steerable** — a persistent lexical motif linking product features to model capabilities: > *"transparent and steerable personalization where the user is always fully in control"* > *"it's fully steable and promptable by uh by users"* > *"language steerable personalized recommendations"* - **Scale numbers as rhetorical anchors.** The speaker repeatedly grounds claims in concrete figures: *"760 million uh active uh users monthly," "184 markets," "100 million uh music tracks," "10 billion playlists," "one in four US premium subscribers."* - **Hedged self-deprecation once** (isolated, not a pattern): *"We're not great at naming these internal things."* ## Teaching Habits - **Explicit roadmap signposting.** Both speakers announce structure before content: > *"I'm going to talk about a little bit of history, the new phase that we're entering and then Jackie is going to help us uh go into the guts of the modes as well."* > *"So let's dive a little bit deeper and talk about how these models are actually trained."* - **"So" as a pervasive discourse marker.** Nearly every explanation and slide transition begins with "So" (*"So how is this model actually trained?", "So what is meaningful about this?"*). - **Rhetorical questions to pivot sections.** A consistent teaching device, especially in the second half: > *"So how much of a difference does this four-stage training paradigm actually make?"* > *"So how is this model actually trained?"* - **Numbered/enumerated structure for complex content.** The NEO paradigm is taught as explicit stages: *"The first I already covered... The second stage we call domain grounding... The third stage we call capability induction... and then there is an optional fourth stage."* - **Pairing concept with concrete example.** Abstract claims are immediately illustrated — e.g., the taste profile concept is followed by a personal anecdote: *"one of my edit is that all Disney music are my kids because we have a bunch of shared devices at home."* ## Handoff and Collaboration Pattern - **Named, graceful handoff:** *"On that note, I'm going to hand it over to Jackie to talk to us about what's one of the core component that underpins this whole system."* - **Complementary roles are stated explicitly** — speaker one covers product/strategy, speaker two covers training mechanics (*"I want to talk about a little bit of history... and then Jackie is going to help us uh go into the guts"*). ## Communication Patterns in the Technical Segment - **Claim → evidence → caveat structure.** Ablation findings are stated, quantified, then bounded: > *"We saw that overall it increased alignment by 5% but on ambiguous queries it actually increased alignment by 91%."* - **Generalizability caveats offered proactively:** *"these abilations were done with Quen, but we also validated that these findings hold with Llama. So, it is not specific to the model backbone."* - **Trade-off framing.** Engineering decisions are presented as explicit trade-offs rather than pure wins: > *"although beam search is a little more latency intensive, we decided that trade-off worked for us."* - **Closing loop-back to the title.** The talk ends by echoing its own framing: *"this is how we taught openweight LLMs to speak Spotify. Thank you."* ## Stability Assessment - **Stable-looking patterns** (multiple occurrences): "So" transitions, rhetorical questions as pivots, scale-number grounding, "steerable" vocabulary, claim-evidence-caveat structure in the technical half. - **Isolated moments** (insufficient repetition to call patterns): the self-deprecating naming comment, the Disney-music anecdote, and the opening thanks to the organizer each occur once. The Disney anecdote also carries a light humorous register not sustained elsewhere, so it should not be treated as a general humor pattern. - **Uncertainty note:** the two speakers differ noticeably — the first half is more product-narrative with anecdotes; the second is more structured and quantitative. Any delivery pattern described above as shared may actually be concentrated in one half of the talk. # Teaching LLMs to Speak Spotify — Yves Raimond & Jacqueline Wood, Spotify ## Speakers - **Yves Raimond** — joined Spotify one year prior; previously worked on personalization at Google Search and Netflix. - **Jacqueline ("Jackie") Wood** — Staff Machine Learning Engineer at Spotify. --- ## Yves Raimond: Framing the Problem and the Shift to Generative Personalization ### Scale of Spotify's personalization problem - Spotify has roughly **760 million monthly active users** across **184 markets**. - Catalog includes **100+ million music tracks**, plus videos, podcasts, and audiobooks — making the **matching problem** "surprisingly complicated." ### Three historical phases of personalization 1. **Curation** — manually assembled playlists targeting specific tastes; still common (~10 billion playlists, many created hourly). 2. **Recommendations** — turning creation/ordering signals into scalable recommendations (e.g., **Discover Weekly**, launched 2014). 3. **Generative personalization** — the current phase: not only matching users to content, but **generating an experience interactively and dynamically shaped around each user**. ### Two conceptual shifts (as stated by Yves) - From **"personalization as guessing"** (ranking algorithms outputting a ranked list) to **"personalization as reasoning"** — systems that can introspect results and judge whether a match is right for a user in context. - From **"blackbox algorithms"** (hard-to-introspect multi-stage ranking systems) to **"transparent and steerable personalization"** where "the user is always fully in control" — giving models the ability to "speak and understand English." ### Product examples of generative personalization - **Spotify DJ** — personalized music playback; since last year users can tap a button to steer the session in real time via voice. - **Prompted playlists** — "full unfettered access to the recommendation algorithm"; supports high-level prompts (e.g., "bands playing in San Francisco tonight") and detailed ones (e.g., a run playlist), with the experience dynamically shaped across phases of the run. - **Taste profile** (launched in New Zealand, coming to more markets) — users can introspect and **edit in natural language** what the algorithm understands about them (Yves's example: editing that Disney music reflects his kids on shared devices, not his taste), and express **aspirational goals** (new genres, topics, languages). - **Personal podcast** (announced recently) — the system generates content, not just recommendations, e.g., a **daily brief**. ### The Large Taste Model - One internal system ("we're not great at naming these internal things") powers all the above applications. - Stated properties: understands every historical interaction and piece of content on Spotify; combines **prediction and reasoning**; fully **steerable and promptable** in real time. - Adoption and impact claims: **~1 in 4 US premium subscribers** interact with it daily; deployment across existing surfaces produced gains on **autoplay, podcast discovery, and DJ message interactions** — "pretty sizable gains across the board." --- ## Jacqueline Wood: Model Architecture and Training (NEO) ### Semantic IDs - **Semantic IDs** embed open-weight LLMs with knowledge of Spotify's catalog: existing content embeddings (e.g., podcast episode embeddings) are quantized into **discrete tokens**; the tokens are added to an open-weight LLM's vocabulary (e.g., **Qwen**), then fine-tuned to understand both natural language and semantic IDs. - Example flow: user asks in natural language for a podcast on morality; the prompt includes listening history as semantic IDs; the model returns a semantic ID plus a **natural-language explanation**. ### NEO: four-stage training paradigm (published paper) 1. **Semantic foundation** — build semantic ID tokens, add them to the LLM vocabulary. 2. **Domain grounding** — align new semantic ID embeddings in the original language embedding space via a **bidirectional mapping** (semantic IDs ↔ text); the **LLM backbone is frozen**, training only new embeddings — mitigating **catastrophic forgetting**. 3. **Capability induction** — **multitask instruction tuning** on Spotify tasks (next-item recommendation, retrieval, etc.); all weights unfrozen; full-parameter or LoRA fine-tuning. 4. **Optional post-training** — e.g., RL fine-tuning. ### Ablation findings - **Multitask vs. single task:** the multitask model consistently **matched or beat** single-task models, indicating positive cross-learning — especially for **audiobook recommendations** (a newer content type), suggesting multitask training helps with **cold-start entities**. - **Training recipe ablations:** dropping domain grounding, or merging it with capability induction, **degraded performance**; the **biggest drop** came from using a **randomly initialized backbone** instead of a pre-trained open-weight LLM. - **Continuous pre-training for domain grounding:** minimal task-specific degradation, but natural-language and world-knowledge capability of the backbone went **"to essentially zero"** — whereas frozen-backbone grounding retains core language ability. - These ablations used **Qwen** but findings were **validated with Llama** — Jacqueline attributes results to the training paradigm, not a specific backbone. ### Inference strategy findings - **Beam search without constrained decoding** generated valid semantic IDs **98% of the time**; constrained decoding adds latency but helps target specific content types (e.g., only new content). - **Top-P sampling significantly hurt accuracy**; the team accepted beam search's latency cost as the right trade-off. ### Novelty and production claims - Prior work exists on generative semantic ID retrieval, tool-based LLM recommenders, and the PUMAS paper, but Jacqueline claims **NEO is "the first example"** combining: grounded catalog items, natural-language steerability, search/recommendation/explanation, and **low-latency, tool-free inference at industrial scale** — and it is **in production today**. - Podcast discovery paper claim: the model can **"break users out of their habitual patterns"** toward unfamiliar content, with "huge wins online." ### Evaluation of generative, explanatory recommendations - Traditional offline metrics are **"not sufficient"**: they measure interaction but not whether a recommendation makes sense, whether explanations are accurate, or whether results align with user intent. - Position: LLM judges are valuable, but require **grounding in meaningful data** to align with human preferences. - **Textual user profiles** as grounding for podcast recommendation judging: **75% alignment** with human preferences. - **Behavioral signals** (similar queries and past interactions) for search judging: **+5% overall alignment**, but **+91% on ambiguous queries** — "especially in ambiguous cases where LLM judges tend to struggle." - **Cranfield-style collections**: human ranking is expensive; grounded LLM judges scaled this up with **0.87 agreement** with human rankings. --- ## Concluding Claim (Jacqueline) > "There is a new era of personalization among us, this generative personalization. And if you want to power language-steerable personalized recommendations for your user, this is how we taught open-weight LLMs to speak Spotify." --- ## Notes on Traceability - All quantitative claims (user counts, alignment percentages, adoption figures, gains) are as **stated by the speakers** on stage; no independent verification is offered in the transcript. - The claim that NEO is "the first example" of combining these capabilities is Jacqueline's characterization, which she herself contrasts against named prior work (semantic ID retrieval, tool-based LLM recommenders, PUMAS). # Video Attribution & Provenance Record **Video ID:** `01a15ca7-2dce-7bc6-9f19-50b91540a8b2` **Title:** Teaching LLMs to Speak Spotify — Yves Raimond & Jacqueline Wood, Spotify --- ## Speakers & Roles | Speaker | Role (as stated in transcript) | |---|---| | **Yves Raimond** (implied first speaker) | Spotify employee (joined ~1 year prior); previously Google (personalization for Search), prior to that Netflix | | **Jackie / Jacqueline Wood** | Staff Machine Learning Engineer at Spotify | - **Role boundary:** Yves covers history, product examples, and the "large taste model" overview; Jackie covers model training (semantic IDs, NEO), ablations, inference, and evaluation. - **Third-party reference:** Speaker thanks **"Deven"** for the invitation and mentions "Deanch" (likely same person) showed a demo earlier — apparently a host/emcee of the event; identity and role otherwise unspecified. - This appears to be a **conference talk** (references to "the previous talk" and "all the talks today"), though the event name is not stated in the source. ## Referenced Sources & Cited Materials - **NEO paper** — Jackie references "a paper linked here" describing the four-stage training paradigm (not named or linked in transcript). - **Podcast discovery paper** — "a paper linked as well here" for production use in podcast discovery. - **Related work named (not attributed in detail):** generative semantic ID retrieval, tool-based LLM recommenders, "the plum paper" (phonetic; likely "PLOME"-style or similar — transcription uncertain). - **Model backbones cited:** Qwen ("Quen" in transcript) and Llama, both described as open-weight. ## Third-Party / Company Claims (recorded, not evaluated) - Spotify scale figures: ~760M monthly active users, 184 markets, 100M+ tracks, ~10 billion playlists. - Spotify DJ steering feature; prompted playlists; **taste profile** (launched in New Zealand); **personal podcast** / daily brief ("announced very recently"). - ~1 in 4 US premium subscribers interact daily with the large taste model (speaker's claim). - Reported gains: autoplay, podcast discovery, DJ message interactions. - Evaluation figures claimed: 75% LLM-judge/human alignment (podcast recs), +5% alignment overall and +91% on ambiguous queries (grounded search judges), 0.87 agreement for Cranfield-style collections, 98% valid semantic ID generation without constrained decoding. ## Promotional Segments - ⚠️ **Flagged:** The talk is substantially **promotional for Spotify products and research** (DJ, prompted playlists, taste profile, personal podcast, NEO). Product launches and metric claims are presented by Spotify employees about Spotify systems — attribute all such claims to the speakers, not to independent verification. ## Availability & Limits - **Papers:** Referenced as "linked" (slides), but no URLs present in transcript — links unavailable from source. - **Q&A:** Not included; transcript ends after closing remarks and music. - **Visuals:** Demos and ablation tables are referenced ("shown here," "rows A and B") but slide content is not in the source — specific data points from slides cannot be verified. ## Uncertainty Notes - First speaker is not named within the transcript itself; attribution to Yves Raimond rests on the video title. - "Quen" / "Deanch" / "abilations" treated as transcription artifacts (Qwen, Deven/Deann, ablations). Welcome everyone. Um really glad and thank you Deven for inviting us as well to talk today. Uh so um uh Jackie and I are going to present today about making LLMs speak Spotify and how we turned our recommendation system on its head to be LLM native. Uh just a few quick words about myself. Uh I joined Spotify a year ago. Before that I was at Google working on personalization for Google search and prior to that I was uh working on personalization at Netflix. Um all right so let's dive in. Um first a couple of numbers to describe the scale of the problem that we have to solve. Um Spotify has about I was a little bit sad not to see it on the DAU by chart because it should be up there but it has about 760 million uh active uh users monthly uh in about 184 markets. But one thing that makes uh the Spotify personalization problem particularly challenging is the size of its catalog. Uh so of course you know Spotify has basically all music ever published a bit more than 100 million uh music tracks but it also has a range of videos, podcasts and audio books as well. So the matching problem uh is uh actually surprisingly uh complicated. So today what we're going to talk about Jackie and I is basically the extent of this matching problem and how we uh how we are solving it. I'm going to talk about a little bit of history, the new phase that we're entering and then Jackie is going to help us uh go into the guts of the modes as well to understand how those things are trained as well. Uh so a little bit on history. Uh initially Spotify personalization was really built around curation. So uh people basically manually assembling playlists that target specific taste. And that's still a very common uh use case on Spotify. Right now we have about 10 billion playlists with many many uh created every every hour. Uh then you know Spotify moved into taking these creation signals and order signals and moving into recommendations. So basically being able to turn these uh creation signals to something that can be applied at scale. Uh and a great example of that would be discover weekly for example that was launched in 2014 as one of the one of the early like recommendation use case on on Spotify. But the phase that we entering now which uh uh which we're going to talk about in more details is something that we call generative personalization where we are not only solving a matching problem from the user to the content. We're also solving the ability to generate an experience that is interactively and dynamically shaped around each user. And that transition from recommendations to generative personalization involves a couple of different uh a couple of different big shifts. One is move from moving moving from personalization as guessing uh where basically you have a ranking algorithms that spits out a brand ranked list of entities in your catalog to personalization as reasoning that can introspect these results and really try to understand whether that's indeed the right match for this user in this particular context. The other aspect as well is moving from blackbox algorithms. uh it's very difficult to fully introspect a multi-stage ranking system for example to transparent and steerable personalization where the user is always fully in control. So uh basically giving the ability to these mods to speak and understand English. The other thing that these these systems can do is uh uh not stopping just at recommending but also generation of experiences and explaining as well. uh and we're going to show a couple of examples of that. So one example that uh we launched a couple of years ago uh so pretty early in that journey was the Spotify DJ. Uh so the Spotify DJ is something that you can spin up that will start playing music for you of course personalized uh and but one interesting thing about it is that since last year you can tap that button on the bottom uh right and steer it in whatever direction you see fit. So at any point you can chime in and let the algorithm know what you want and it's going to steer the session in the in in that in that direction. Uh another example of what we call generative personalization is uh showcased in a prompted playlist here that Deanch showed a little bit earlier as well. Uh and here you can you basically have full unfettered access to the recommendation algorithm that Spotify has. uh and you can uh prompt it with very high level prompts or very detailed prompts. On the left hand side here, I have a prompt that tells me uh create me a playlist of bands that are playing in San Francisco tonight. Uh so turns out there's a bunch of good shows if you're excited to check them out. On the right hand side, you see a prompt that is asking for a playlist to accompany me on my run. And what's interesting with the right hand side as well is that you will see that the experience itself gets dynamically shaped as a function of the request on the user to be able to introduce itself and different phases in my run as well. Uh, another one that I'm really excited about, we launched it in New Zealand a couple of months ago and uh, it's coming soon in more markets is something called the taste profile. And that basically gives you the ability to introspect in natural language what the Spotify algorithm has understood about you in a way that you can edit and refine. Uh, so if you see something that's missing or something that's wrong. Uh so for example one of my edit is that all Disney music are my kids because we have a bunch of shared devices at home but please don't recommend that to me that's not my taste please. Uh and the algorithm would then take that into account uh making sure that we never recommend this this in the wrong context. Uh and similarly you can also use the taste profile to share some more aspiration uh goals as well. Uh so getting into a new genre, getting into a new topic, uh learning about a new language for example, all of these things can be can be done. Uh another one that's coming uh soon uh which we uh announced very recently is something called personal podcast where the the the generative personalization system doesn't stop at just recommending and ascending experiences but also generating content as well. In this particular example, I'm generating a daily brief that's uh that's generated on a cadence daily. Uh and I'm going to uh let it uh tell me about what's happening in my community. All right. So now to go into the guts of it. So there's one big system that controls like all of these different applications I mentioned and we call that internally some the large taste model. We're not great at naming these internal things. Um so it has a couple of properties. One is that it understands every historical interaction piece of content on Spotify. Uh it combines prediction and reasoning to the point that I mentioned earlier. So not only guessing but also reasoning layered on top and it gives users the ability to shape and generate experiences in real time. So it's fully steable and promptable by uh by users. Um and uh as of today about one in four US premium subscribers interact with that system on a daily basis as well. So that's pretty exciting. Um, one thing that's exciting as well to uh is that deploying this system across existing recommendation surfaces as well also led to some gains. Uh, we saw gains on autoplay, we saw gains on podcast discoveries. Uh, we saw gains on users interacting with DJ messages as well. So, we saw pretty pretty sizable gains across the board as well by deploying the system. On that note, I'm going to hand it over to Jackie to talk to us about what's one of the core component that underpins this whole system. Hi everyone, I'm Jackie or Jacqueline, a staff machine learning engineer at Spotify. So let's dive a little bit deeper and talk about how these models are actually trained at least at Spotify. So semantic IDs were presented in the previous talk, but that is how we are embedding these openweight LLMs with knowledge of Spotify's catalog. They are created by taking existing content embeddings such as podcast episode embeddings and applying a quantization algorithm to convert them to a set of discrete tokens. We then take a openweight LLM such as Quen and we modify its vocabulary to add these new special tokens and then we fine-tune the model to be able to understand both natural language as well as these new special tokens semantic IDs that represent Spotify catalog entities. So we can power experiences such as this where the user can ask in natural language for a podcast on morality. And that is passed to the prompt along with their listening history represented as semantic IDs. And the model responds both with a relevant semantic ID podcast episode as well as a natural language description of why they recommended that to this user. So how is this model actually trained? Um we published a paper linked here uh describing our training paradigm called NEO which consists of four distinct stages. The first I already covered which is the semantic foundation stage where we construct meaningful semantic ID tokens and then add them to an openway LLM's vocabulary. The second stage we call domain grounding in which we align these new semantic ID token embeddings in the original language embedding space. We do this by learning a birectional mapping between semantic ids to text, text to semantic ids and any combination. And we actually freeze the LLM backbone at this stage and only train the new semantic ID embeddings. So the original model weights and embeddings are frozen and we just learn those new semantic ID tokens and this helps us to mitigate catastrophic forgetting of the pre-trained LLM's core language abilities. The third stage we call capability induction which is multitask instruction tuning on tasks that Spotify cares about such as the ones shown here. next item recommendation retrieval etc. This is done by unfreezing the whole model all of its weights and embeddings and running either full parameter fine-tuning or Laura fine-tuning on the multiple Spotify tasks and then there is an optional fourth stage to do post- training such as RL fine-tuning etc. So, how much of a difference does this four-stage training paradigm actually make? I'll dive into a few of the abilations we've done to investigate this. First, we assessed whether multitask training is actually hurting performance by comparing the multitask model against single task variance. And we consistently saw that across our tasks, the multitask model can match or actually beat the single task performance indicating that there is some positive cross-learning happening across the tasks. This is particularly noticeable for audiobook recommendations. If you look here, which is a new newer content type at Spotify, demonstrating that these multitask models can help with cold start entities by learning from other items in the catalog such as podcast recommendations, how to make meaningful audiobook recommendations. Then we did some abilations on the actual training recipe. We evaluated both dropping the frozen backbone domain grounding stage altogether as well as combining the domain grounding stage with the capability induction stage in a multitask instruction tuning stage that those are rows A and B here in the middle and we saw for both of those that it degraded performance but actually the biggest drop in performance was from using a randomly initialized backbone instead of the pre-trained openweight LLM that we are using. We also evaluated using continuous pre-training for the domain grounding stage. And as you can see, it's it's minimal actual degradation on the task specific performance. But where continuous pre-training really hits us is on the natural language and world knowledge capabilities of the pre-trained backbone LLM we are using. after we do continuous pre-training, it goes to essentially zero versus if we do the frozen backbone domain grounding. We retain all of that core language ability and are still able to learn the semantic ids. I want to call out that these abilations were done with Quen, but we also validated that these findings hold with Llama. So, it is not specific to the model backbone, but actually the training paradigm itself. And then lastly, we did some investigation on different inference strategies and their effect on accuracy versus latency. We tested beam search with both constrained decoding and not. And we saw that even without constrained decoding, we can generate valid semantic IDs 98% of the time. Constrained decoding does add a little latency overhead, but it's also helpful for specific cases where you want to target specific types of content, such as only make new content recommendations, for example. We also compared top P sampling to beam search and saw that top P sampling pretty significantly hurts our accuracy. So although beam search is a little more latency intensive, we decided that trade-off worked for us. So what is meaningful about this? There have been lots of work in the industry in the space on generative semantic ID retrieval, toolbased LLM recommenders, the plum paper, etc. But NEO is actually the first example of combining all these capabilities into one system that understands grounded catalog items, is naturally language steerable, can do search, recommendation, explanation use cases, as well as low latency tool-free inference at an industrial scale. And we are using this in production today. Um there's a paper linked as well here for how we're using this to power podcast discovery. What we saw is that using a model train like this, we can break users out of their habitual patterns and get them to listen to more unfamiliar content. And we saw huge wins online with this. So none of this works without meaningful evaluations. So I want to talk about that a little bit as our recommendations are becoming more generative and explanatory. We our original maybe traditional offline eval metrics are not sufficient. Yes, they can tell us whether or not the user interacted with that content, but they can't tell us whether or not the user or that recommendation makes sense for the user, whether the explanation is accurate, whether it aligns with the user's intent, etc. So, that's where LLM judges really shine. But in our work at Spotify, we really find that you need to invest in grounding your LLM judges in meaningful data. so that they can be reliable evaluators that align with human preferences. So, a few examples um for evaluating the podcast recommendations that I just mentioned, we create textual user profiles summarizing the users's listening history and that is passed to the LLM as a judge and we saw that this corresponded with a 75% alignment between the LLM judge and human preferences. Similarly, you can use actual behavioral signals to ground these LLM judges. So, for example, for a search task, you can for a given query, you can take similar queries and how the user has interacted with them in the past and pass that to the model. And we saw that overall it increased alignment by 5% but on ambiguous queries it actually increased alignment by 91%. Showcasing the value these this grounding of the LLM judge plays especially in ambiguous cases where LLM judges tend to struggle. Lastly, we used grounded LM judges to scale up our Cranfield style collections. These are evaluation sets that are constructed by taking candidates from multiple different sources, creating a pool, and then using a human to rank that pool. However, that human ranking stage is expensive. So, we invested in significant grounding for our LM as judge and are able to have an LLM judge that aligns with uh human system rankings with an agreement value of 0.87. In summary, um like we're going to hear a lot about today in all the talks, there is a new era of personalization among us. this generative personalization. And if you want to power language steerable personalized recommendations for your user, this is how we taught openweight LLMs to speak Spotify. Thank you. \>\> \[music\]