Philip Kiely (Baseten) surveys recent developments in inference engineering, framing two distinct contexts: local inference (prioritizing getting models working, then improving quality) and data center inference (prioritizing throughput). Topics include the TurboQuant moment and why 4-bit KV caching helps memory-constrained machines but hurts data center tokens-per-second; quantization (4-bit weights as the data center answer); KV cache compaction via a learned bottleneck that cross-attends fixed query vectors; and speculation, where drafter quality evolved from same-family small models to models trained on target hidden states, a diffusion drafter proposing 8–16 tokens at once with tripled acceptance in production, and continuous speculator retraining on live prompts that can double acceptance given storage, compute, and permissions. The talk also covers how training-for-inference is blurring the line between training and inference. # Video Identity Observations **Video:** *What's New in Inference Engineering — Philip Kiely, Baseten* **Source evidence:** title and transcript only (no curated metadata supplied) ## Speaker & Channel Self-Description (Explicit Claims) - **Author identity:** The speaker identifies himself as Philip and anchors his credibility in authorship: *"I'm here because I wrote a book... I wrote this book called Inference Engineering. We published it three or four months ago."* - **Publisher affiliation:** He repeatedly references **Baseten** ("our model performance team," "base 10 research team," "come down with me to the base 10 booth"), indicating a company-affiliated technical channel rather than an independent personal one. - **Reach metrics claimed:** >11,000 paper copies, ~30,000 digital copies, "24 million Twitter accounts... have seen something out of inference engineering." - **Community standing:** *"This is my third year at the AI engineer worlds fair... It's the highlight of the calendar every single year."* ## Intended Audience - **Inference/ML infrastructure engineers**, particularly the data-center side: *"This talk [is] going to be focused on advances in data center oriented inference engineering because that's what I know."* - Assumes familiarity with jargon without extended hand-holding: **TTFT, TPS, KV cache, speculative decoding, disaggregation, NVFP4, Eagle 3, B200** — used with only brief review segments. - Practitioners making production decisions: framed around *"the practical day-to-day of how do I make X model faster."* - Secondary acknowledgment of local-inference hobbyists, though positioned as outsiders to the talk's scope: *"there's a lot of really cool stuff happening in the local world as well."* ## Claimed Scope - A **post-publication update** to a defined body of work: *"the first public addendum of all new information since the book came out"* (Feb 23, 2026 reference). - Organized around what he calls **"the big three"** inference techniques — **quantization, caching (KV), speculation** — plus trend forecasting: *"a little bit of prognosticating... what I'm excited about in inference."* - A deliberate thesis the channel promotes: **"training for inference"** — *"many optimizations for inference come from a dedicated training process... the lines between training and inference are getting blurrier."* - Positioning on a **two-world taxonomy**: local inference ("get it working, then make it less dumb") vs. data center ("get it working, then make it less slow"). ## Demonstrated Presentation Patterns (Observable, Not Claimed) - **Vendor evaluation voice:** Presents internal research findings on third-party techniques (e.g., TurboQuant rejected for production due to TPS halving, but endorsed for local users; **dFlash** "up and running in production"; **dSpark** "no production results yet"). - **Team attribution habit:** Regularly credits named colleagues (*"shout out... Ali from our model performance team," "Charlie and Mudith from our post-training team"*) and points to external artifacts (YouTube chalk talks, Twitter threads). - **Outbound funnel:** Closes with a free PDF/book giveaway at the Baseten booth and social handles — content serving as a lead-in to the company's presence. - **Hedged forecasting:** *"The following is like personal opinion and speculation and public information... if I knew anything that was actually coming out, I wouldn't be able to talk about it."* ## Notes for Synthesis - All evidence is from the transcript; the video title additionally confirms speaker–company attribution ("Philip Kiely, Baseten"). - No claims about broader channel content (upload cadence, other formats) can be made from this single talk. # Delivery Observation — Philip Kiely, "What's New in Inference Engineering" ## Overall Delivery Style - **Conversational, talk-track register.** Frequent fillers and self-corrections ("Um", "you know", "uh", "So") give the delivery an informal, spoken feel rather than a scripted one. - **Direct audience address.** Repeated second-person framing: *"So, let's get started"*, *"Let's talk about it. Let's talk about KV compaction"*, *"So what is next in inference?"* - **Rhetorical question pivots.** Uses questions to launch sections: > "Why in the world would you do this? Like why would you write a book about something that's changing so fast?" > "But what if there was a middle load? What if there was a way where..." ## Recurring Phrases & Verbal Habits - **"You know"** — pervasive connective throughout ("you know, great reception", "you know, I believe that...", "you know, I thought I was done"). - **"Quick review"** — a consistent on-ramp for background sections: > "just as a quick review, quantization is when..." > "Let's talk about KV compaction. Again quick review." > "as a review, uh speculative decoding..." - **Colloquial asides / humor.** *"and you just kind of like keep doing that until you're super rich"*, *"it like made the memory stock macro dip... uh which which was wrong but anyway"*, *"I do not unfortunately have the time or the genius to explain everything up here"*. - **Casual numerics.** Prefers approximate quantifiers: *"20 million people saw Toboquant"*, *"two or four times slower"*, *"eight or 16 tokens"*, *"20% to even 2x improvement"*. ## Teaching Habits - **Preview-then-deliver structure.** Opens with an explicit roadmap: *"We're going to talk about what happened to Turboquant. Talk a little bit about KV compaction. We're going to spend a lot of time on deflash..."* — then executes it nearly in that order. - **Review-before-new-material.** Each major section opens with a one-paragraph refresher of the concept before introducing the news (quantization, KV cache, speculative decoding all follow this pattern). - **Trade-off framing.** Consistently pairs benefits with drawbacks: > "the drawback is pretty big um for turboquant... it cuts TPS by more than half and that's just an unacceptable trade-off" - **Contrast framing of two worlds.** Develops a local-vs-data-center dichotomy with parallel formulas: *"first you get it working and then you make it less dumb"* vs. *"get it working and then make it less slow"*. - **Explicit boundary-labelling of claims.** Separates opinion from fact: *"the following is like personal opinion and speculation and public information"*, and flags low-confidence items: *"DSpark is is new research. Um so this one I can basically only say like hey it exists."* ## Attribution & Credibility Patterns - **Names individual teammates.** *"shout out Waterlue intern on Twitter. Um Ali from our model performance team"*, *"Charlie and Mudith from our post training team"*, with redirects to external resources ("chalk talk at Kosa compile... it's up on YouTube"). - **Grounds claims in production data.** Repeatedly anchors to deployment status: *"that's two of the techniques"*; *"Dlash we do have up and running in production"*, *"we don't have any production results with it yet to share"*. - **Book as recurring anchor.** References the book multiple times (publication date February 23rd, sales figures, free copies at the booth) and frames the talk as *"the first public addendum of all new information since the book came out"*. ## Structural Patterns - **Signpost transitions.** *"So that's two of the techniques... The final one is speculation."*, *"So, let's get started"*, *"So what is next in inference?"* - **Thematic thesis restated.** "Training for inference" is named as a theme, demonstrated in the KV compaction and speculation sections, and re-invoked in the outlook: *"the theme of training for inference is going to be something that continues to have a big impact"*. - **Forward-looking close.** Ends with prognostication tied to hardware cycles ("Ampio... Hopper... Blackwell", upcoming "Ruben"), then a call to action (free PDF via QR code, booth visit). ## Stability Assessment - **Well-supported patterns** (multiple occurrences): "you know", "quick review" scaffolding, review-then-news section structure, production-results grounding, team member attribution. - **Thinner evidence** (single occurrences; treat cautiously): the "super rich" humor, the memory-stock anecdote, and the hardware-cycle history appear only once each and should not be treated as established habits. # Observed Positions & Claims — "What's New in Inference Engineering" (Philip Kiely, Baseten) ## Speaker & Context - **Speaker:** Philip Kiely (identified via video title and first-person introduction), speaking at the AI Engineer World's Fair. He identifies as a Baseten employee via references to "our team," "the Baseten research team," and "the Baseten booth." - **Self-described framing:** This talk is presented as "the first public addendum of all new information" since his book *Inference Engineering* (published, per his account, February 23rd; he states more than 11,000 paper copies and approaching 30,000 digital copies distributed). ## Core Framing Claims ### Two types of inference engineering - **Local inference:** strategy is "get it working" on available hardware by "squishing the model with quantization, distillation, pruning," then "make it less dumb" — restoring baseline intelligence at batch size one after compression-induced damage. - **Data center / batch-size world (speaker's own domain):** get a build working at day zero, then "make it less slow" via KV-aware routing, speculation, and disaggregation. - He states the two worlds "have a lot to learn from each other," and that this talk focuses on the data center side. ### Blurring training/inference boundary - In the book he "generally assume[s] that the weights are a finished product," but now reports that **many inference optimizations come from dedicated training processes**, making the training/inference lines "blurrier and blurrier." - He describes a cycle: faster inference → more data → better models → faster inference. ### The "big three" techniques - Quantization, caching (KV cache), and speculation are identified as what practitioners "reach for" day-to-day to make models faster. --- ## Quantization ### Turboquant - **Claim:** Turboquant is a quantization approach popularized in March (of the year in question) using **polar coordinates**, allowing KV cache quantization to **4 bits** — half the memory and bandwidth, "effectively double the bandwidth" when moving KV cache through system memory. - **Claim:** He states ~20 million people saw Turboquant content and that it "made the memory stock macro dip" briefly — a reaction he characterizes as "wrong." - **Key drawback (his position):** Turboquant requires **additional forward-pass computation during decode, cutting TPS by more than half** — "an unacceptable trade-off for a lot of the production use cases." - **Position:** Baseten took "a good hard look" at Turboquant but is **not using it for real workloads**, remaining on **NVFP4 quantization** with a focus on weights rather than KV cache. - **Qualification:** He concedes Turboquant "is a fantastic research paper" and **well-suited to local inference**, where memory is the primary bottleneck and extra compute matters less. - **Attribution note:** The underlying math analysis is credited to "Ali from our model performance team" (referred to as a "Waterlue intern" on Twitter; the speaker is "not sure if he's still an intern"). ### Current Baseten focus - NVFP4 with focus on **weights over KV cache**, avoiding flattened probability distributions; KV-aware routing, KV offloading, KV sharing (NIXL, NVIDIA Dynamo); CPU/host-memory offloading. - **Quantization across modalities:** extending NVFP4 benefits to image and video models. --- ## KV Caching & Compaction ### Review framing - KV cache reuse makes systems faster; described as one of the few sources of **"lossless memory,"** scaling linearly with input data. - Context-compression techniques (agent harness compression, RAG, writing to files) scale sublinearly — he poses the question of a middle ground: significant compression with "near lossless information retention." ### Existing approaches - Cites papers "attention matching" and "cartridges" as promising, high-compression-ratio methods — but notes both **run at inference time**. ### "Still" (transcript rendering; likely a product/research name) - From the Baseten research team; presented as an example of **training for inference**. - **Stated mechanism:** a perceptual bottleneck using a fixed set of learned query vectors that cross-attends against the full KV cache and "produces a set of compact keys and values in a single forward pass," creating "a fast differentiable compressed memory that the LLM can attend to as if it was real context." - Credits Charlie and Mudith (Baseten post-training team) for a chalk talk; the speaker explicitly declines to fully explain it: "I do not unfortunately have the time or the genius to explain everything up here." --- ## Speculation ### Review and lineage (attributed to the book / industry history) - Speculative decoding: draft tokens verified in the forward pass; **fully lossless**; improves tokens per second. - Progression he describes: 1. **Classic spec decode** — small same-family models as drafters; "small models are like not great draft token generators." 2. **Medusa** — added decoder heads. 3. **Eagle 3** — a billion-parameter model trained on the target model's hidden states; per the speaker, the best method as of roughly February. ### DFlash (transcript rendering: "Dlash"/"deflash") - **Diffusion for speculation:** generates a **sequence** of draft tokens rather than one, analogous to diffusion image/video generation. - **Trade-off as stated:** DFlash models "might be two or four times slower to run" but predict **eight or 16 tokens at once** vs. Eagle's one — so "a single DFlash forward pass is faster than the entire Eagle draft phase." - **Claim:** Cross-attention among draft tokens yields higher acceptance rates ("in speculation acceptance rate is everything"). - **Production claim:** "in the wild we're seeing a more than 3x improvement," measured on a single B200 with Qwen 3B (transcript: "quen 38B" — ambiguous rendering). - **Technical detail given:** trained with an attention mask enabling **bidirectional drafting** while enforcing causal consistency via "clean tokens" sampled per block. ### DSpark (transcript rendering) - Released "a couple days ago" per the speaker; pairs the diffusion speculator with a sequential model to improve acceptance weights. - **Explicit limitation:** "I can basically only say like hey it exists... we don't have any production results with it yet to share." ### Continuous speculator retraining - **Claim:** Continuously retraining the speculator on live prompts/responses yields **20% to 2x improvement in token acceptance rates**. - **Stated difficulties:** heavy storage and compute requirements; **permission to use the data** must be secured; changing the underlying model requires changing the speculator. - **Forecast:** worthwhile optimization "for very very large scale systems." --- ## Forward-Looking Claims (explicitly flagged as opinion) - He prefaces: "The following is like personal opinion and speculation and public information and like if I knew anything that was actually coming out, I wouldn't be able to talk about it." - He notes having been through three hardware cycles (Ampere, Hopper, Blackwell) and that software stacks lag chip availability. - Expectations: - **Rubin** hardware will have "fantastic" NVFP4 performance; confidence in NVFP4 (borrowed partly from local inference) will unlock it. - **Disaggregation and system-wide communication** (e.g., PD disaggregation, KV cache movement) will grow in importance. - **Training for inference** will remain a major industry theme. --- ## Contradictions & Qualifications Preserved - Turboquant: praised as "fantastic" research **while rejected** for production data center use — the tension is explicit and retained. - DFlash is in production; **DSpark is not** — the speaker keeps these confidence levels distinct. - The speaker's book claims principles "have been pretty solidified," yet the entire talk documents rapid post-publication change (Turboquant, Still, DFlash, DSpark) — an acknowledged tension in his framing. - Self-deprecating hedges ("I don't have the time or the genius," "24 million people or 24 million Twitter accounts... we'll see how many people that actually is") are preserved as stated uncertainty. ## Traceability Notes - All claims above derive solely from the transcript. Several proper nouns are rendered inconsistently in the transcript (e.g., "Turboquant"/"Toboquant"/"toocquant"; "Dlash"/"deflash"/"deep flash"; "Waterlue"; "Reuben"; "Kosa compile"; "influence"/"influence engineering" for "inference"); both renderings are preserved here where relevant, and no corrections have been invented. # Speaker & Provenance Record ## Video Identification - **Title:** *What's New in Inference Engineering — Philip Kiely, Baseten* - **Video ID:** `01a15ca7-59c9-755a-a04f-74da11612976` ## Speakers & Roles | Speaker | Role | Basis | |---|---|---| | **Philip Kiely** | Presenter, speaking in professional capacity affiliated with **Baseten**; author of *Inference Engineering* | Self-identified in transcript; title metadata | | Unidentified second voice (`>>`) | Not attributable | Transcript contains only a closing marker before music/applause; no speech content | - Kiely states he is a **third-year attendee/speaker** at the AI Engineer Worlds Fair (2024, 2025, present year) and got his start as a speaker/engineer there. - Reference to "our team," "our model performance team," "our post training team" indicates he speaks for Baseten, though his exact job title is **not stated** in the transcript. ## Talk Framing - Described as the **"first public addendum"** of new inference material since his book's publication (stated as **February 23rd** — transcript says "February 23rd of 2026" in one passage and implies February in another; treat year as stated but note internal ambiguity). ## Referenced People & Their Roles - **Ali** — Baseten model performance team; possibly intern; from "Waterlue" (as transcribed); wrote a piece on Turboquant math and Twitter content on cross-modality quantization. - **Charlie and Mudith** — Baseten post-training team; gave a chalk talk at "Kosa compile" (as transcribed); talk reportedly available on **YouTube**. ## Referenced Sources & Third-Party Claims - **Book:** *Inference Engineering* by the speaker — claimed figures: 11,000+ paper copies, ~30,000 digital copies, reach to "24 million" Twitter accounts. *These are speaker claims, not independently verified.* - **Turboquant** — quantization approach using polar coordinates, popularized in March; speaker claims it caused a brief "memory stock macro dip"; attribution to a specific paper/author not given. - **Papers/techniques cited:** attention matching, cartridges, **Medusa**, **Eagle 3**, **DFlash** ("deflash"), **DSpark**, **Still** (Baseten research; KV compaction), NVFP4 quantization, Nvidia Dynamo. - **Production metrics claimed by speaker:** 3x+ improvement from DFlash (measured on single B200, Qwen 38B as transcribed); 20%–2x acceptance-rate gains from continuous speculator retraining. - **Hardware generations:** Ampere, Hopper, Blackwell referenced; **Rubin** ("Reuben" as transcribed) discussed as upcoming. - Prior talk: AI Engineer Miami (quantization talk), "last month." ## Promotional Segments — FLAGGED - **Book promotion:** free PDF via QR code and free physical copies at the **Baseten booth**. - **Company promotion:** repeated references to Baseten research, production results, and booth presence. - Closing appeal to follow the speaker on **Twitter** and **LinkedIn**. ## Availability & Limits - Chalk talk (Charlie & Mudith): reportedly on YouTube — availability stated but no link given. - Speaker explicitly notes forward-looking section is **personal opinion/speculation** and withheld non-public information: *"if I knew anything that was actually coming out, I wouldn't be able to talk about it."* - DSpark: no production results available ("very exciting… we don't have any production results with it yet"). - No comment section availability stated in source material. ## Uncertainty Notes - Several proper nouns are transcription-approximations: "Turboquant/Toboquant/toocquant," "deflash/Dlash/DFlash," "Waterlue," "Kosa compile," "Reuben," "Ampio" (likely Ampere), "influence" (likely "inference"). - Audience Q&A: none present in transcript; video ends with applause and music. - Speaker's exact role/title at Baseten not stated. Um I am here to talk about what is new in inference engineering. So hi I'm Philip and I'm here because I wrote a book. Uh this is my third year at the AI engineer worlds fair. This is my favorite conference in the entire world. It's the highlight of the calendar every single year. I really got my start as a speaker and as an engineer here in 2024. I came back in 2025 and did a bunch of stuff. I'm here again. I love it here and I'm very thankful to the organizers for always having me. Um I wrote this book called Inference Engineering. We published it three or four months ago and I've just been overwhelmed by the response. Uh we've done more than uh I yeah I published it in on February 23rd. Um at this point we've done more than 11,000 paper copies. We're coming up on 30,000 digital copies and 24 million people around the world or 24 million Twitter accounts. So we'll see how many people that actually is um have seen something out of inference engineering. And with all of this, you know, great reception, there has been one question that people have been asking me. Why in the world would you do this? Like why would you write a book about something that's changing so fast? Well, you know, I believe that a lot of the principles of inference engineering at this point have been pretty solidified and there's a lot that we can, you know, learn and and kind of repeat over generation and generation of model. But today I'm here to talk about what's new in inference engineering. This is the first public addendum of all new information since the book came out. We are going to review the inference engineering principles a little bit and then we're going to talk about all the stuff that's happened since February 23rd of 2026 in the inference world. We're going to talk about what happened to Turboquant. Talk a little bit about KV compaction. We're going to spend a lot of time on deflash and some other new exciting things in speculative decoding. And then I'm going to do a little bit of prognosticating, a little bit of forecasting of what I think is going to happen in influence coming up here and what I'm excited about. You know, hopefully being able to talk about next time you guys see me up here. Cool. So, let's get started. So, you know, one thing I've been identifying now out of tons and tons of conversations with people about influence is a handful of of shared principles. And one of the big ones I was on this uh podcast the other day with uh so uh we were talking about inference and you know there's two types of inference engineering that have really emerged. There's local inference where the overwhelming strategy is just get it working on whatever hardware you have by squishing the model with quantization, distillation, pruning, however you can, you know, splitting it across whatever GPUs you happen to have in your house. And first you get it working and then you make it less dumb. you take away whatever you know catastrophic issues all of this compression of the model has created and you try and get it back to that baseline intelligence running at a batch size of one and then there's there's my world which is the batch size and data center world where it's get it working you know just day zero get the build of VLM up get it working and then make it less slow do stuff like KVAware routing speculation disagregation and you know with within these two worlds I think that we a lot to learn from each other. I am in this talk going to be focused on advances in data center oriented inference engineering because that's what I know but there's a lot of really cool stuff happening in the local world as well. So in the book in inference engineering I generally assume that the weights are a finished product and I do think that the handoff from training to inference is an important one to keep in mind and it's a good way of kind of delimiting the space. However, what I've found more and more recently is that many optimizations for inference come from a dedicated training process. And so the lines between training and inference are getting blurriier and blurriier. And that's an interesting thing to keep in mind. We're seeing this cycle where you get faster inference, which gives you more data, which you use to train a better model, which gives you faster inference, which gives you more data, and you just kind of like keep doing that until you're super rich. Uh so with with training for inference uh we have a bunch of new techniques to talk about across what I like to call the big three. So we're going to talk about some news in quantization, some news in caching, specifically the the KV cache mechanism, and some news in speculation because these are, you know, there's a lot of other stuff in the world of inference, including some stuff I'm going to talk about at the end. But when it comes to the practical day-to-day of how do I make X model faster usually these are the three techniques that people are reaching for. So first thing I publish a book it's February I'm feeling awesome about myself. I'm like wow everything you need to know about influence in one place. And then uh we uh we had some news in the quantization world. So just as a quick review, quantization is when we use a smaller, less precise number format in order to save ourselves on bandwidth, save ourselves on compute, make TTFT better, make TPS better. Uh, it's usually kind of hardware specific, gives you cost savings, but potentially degrades model quality a little bit. And by the way, if you want to hear my whole rant about quantization, I did a talk at AI Engineer Miami last month about how quantization is is not necessarily as evil as it sounds and that there's many things you can do to preserve quality through that process. So I was feeling good about my treatment of quantization and then 20 million people saw Toboquant and in fact it like made the memory stock macro dip for for a minute just because everyone was like oh memory is going to be so much more efficient now like we don't need any more flash memory uh which which was wrong but anyway it was this new quantization approach um that was popularized in March of this year that uses polar coordinates for quantization and allows you to quantize the KV cache down to four bits. And it was like super hot and I was like, "Oh man, like there's this whole thing that that I left out." And like what what is this going to look like? And so our team did a bunch of research on this. Um this is if you know uh shout out Waterlue intern on Twitter. Um Ali from our model performance team. Um I'm not sure if he's still an intern actually. Uh but yeah, he is from Waterlue. Um anyway, so he he wrote this great piece about the math behind turboquant. And basically the benefit you get out of turboquant is that you get to represent the KV cache with four bits instead of eight bits. You save half the room and half the band and and you get effectively double the bandwidth when you're moving KV cache around in your system memory. But the drawback is pretty big um for turboquant. Turns out that you need to do additional computation in the forward pass to account for this during decode and it cuts TPS by more than half and that's just an unacceptable trade-off for a lot of the production use cases. So we we took a good hard look at toocquant but but are not using it for you know any any of these real workloads. We're we're still on the traditional NVFP4 quantization. That said it actually is a great technique for the local inference folks. So if you are running a model especially a long context language model on your local computer on GPUs in your basement um you have a very limited amount of memory that's the number one bottleneck and so anything that can free up memory from KV cache and allow you to put those longer sequences on there is going to be very valuable and the additional forward pass computation is going to be like less of a a drawback. Um, so still Turboquant is a fantastic research paper, a really great technique that just ended up not being as applicable in the data center inference world as it might have first appeared. Um, you know, instead we're focused on, you know, NVFP4 with a focus on quantizing the weights versus the KV cache. Um, you know, doing our best to find rough edges in the quantized weights. um make sure that we're not flattening out of probability distributions for the KV cache itself. Focusing instead on KV aware routing, KV offloading, KV sharing using uh you know nickel and using Nvidia Dynamo and other tools in order to move the KV cache around the system and potentially offload to CPU, ordinary memory, etc. um versus trying to use uh to Turboquant to to compress it. Um and then we're also focused on quantization across modalities. Um so thinking about how can we apply the benefits of NVFP4 not only to language models but also to you know image and video models. Ali also wrote a lot of great stuff on Twitter about that which you should check out. So that said the KV cache is still very important. Let's talk about it. Let's talk about KV compaction. Again quick review. KV cache. If you put in the same prompt with the same prefix, uh you get to reuse the tokens that you calculated prefill on last time. That makes your whole system faster and more efficient. Broadly, KV cache is lossless memory. There's only a couple sources of lossless memory. When we think about our inference system, we have the content of the prompt, the context, you have the the KV cache, and that's going to scale linearly with the amount of data you pass in. And now if you're thinking about you know million token sequence lengths uh that actually gets pretty substantial. So a lot of people are thinking about how do you compress memory? How do you compress context? Agent harnesses will compress context. Rag search all these techniques that we've been talking about for years are a sort of compression of a larger context into something that you can give to a model. You can write to files. All of these things scale sublinearly with the amount of data that you have. But what if there was a middle load? What if there was a way where you could get quite a bit of compression in the data that you remembering with near lossless information retention? So we uh you know we have a lot of different ways that we can think of what to keep in the cache. You know recent compaction methods have shown that we can replace the cache with a much shorter one. We've got papers like attention matching and cartridges that have given really promising uh outcomes here with uh high compression ratios. But both of these are run at inference time. Again, one of the techniques I want to talk about or one of the themes I want to talk about is training for inference. So in this case I want to introduce something called still by the base 10 research team where the synthesis on top of the cache where we're keeping a loaned representation of the information rather than a the information directly or a sort of deterministic subset of it is amotized via training. So Charlie and Mudith from our post training team did a fantastic chalk talk at Kosa compile recently. Um it's up on YouTube. So I would encourage you to take a look at it if you're interested in learning about KV compaction. Um I do not unfortunately have the time or the genius to explain everything up here. Um but the the basic mechanism is that still is a perceivable bottleneck um that takes a fixed set of loan query vectors crossends it against the full KV cache and produces a set of compact keys and values in a single forward pass. This creates a fast differentiable compressed memory that the LLM can attend to as if it was real context. So if you're interested in KV compaction, definitely check out Charlie and Mudith's work. Um it's been a really fantastic thing to learn about. So that's two of the techniques. We've talked about quantization. We've talked about caching. The final one is speculation. And there's been a lot of change here. Um as a review, uh speculative decoding, we're going to use draft tokens. We're going to verify them during the forward pass and we're going to use that to generate more than one token per forward pass. It helps a lot with tokens per second and it is a fully lossless optimization which is great because we don't have to worry about quality at all. Now in in the sort of history of speculation we started with speculative decoding. All of these are in the book. Um you have spec where you use a small model from the same family to generate draft tokens. Turns out small models are like not great draft token generators. They're great small models. So we invented as an industry a bunch of new methods like Medusa where maybe just you add decoder heads to the model and then eventually Eagle 3 which was hey what if instead of taking a tiny model from the same family, we actually train a billion parameter model on the hidden states of the target model to generate draft tokens. And that actually worked really well. And so, you know, as of of maybe February last year of of February of this year, Eagle 3 was the best method in speculation. Now, we got Dlash. Dlash is even better. So, it's diffusion for speculation. Uh, Dlash creates a sequence of draft tokens instead of a single token. So the model is a diffusion language model which means it creates a whole sequence of tokens in the same way that a video or image generation creates a sequence of frames or a sequence of pixels and iterates over it rather than doing a auto reggressive token generation. Dlash models might be two or four times slower to run, but they're going to predict eight or 16 tokens at once um in that in that window while Eagle is only doing one at a time. So a single D flash forward pass is faster than the entire eagle draft phrase and predicts more tokens. These tokens are able to cross attend to each other and generally create a higher acceptance rate because in speculation acceptance rate is everything. So in the wild we're seeing a more than 3x improvement uh from deep flash. Um this is measured with a single B200 quen 38B. Um and we can see it versus Eagle. It's a substantial improvement in the tokens uh both the token acceptance rate and the tokens per second. These deflash models are trained with a attention mask for birectional drafting. Um so the target model is going to provide the context. Um and within each block we're going to have a subset of clean tokens that are sampled and the attention mask is going to enforce causal consistency. Uh but it is still going to allow for birectional attention. um where in you know a traditional auto reggressive model you're only looking at the tokens in a single direction. So that's why we're able to you know take advantage of this diffusionbased architecture and then you know I thought I was done and then a couple days ago uh DSpark came out. Now Dlash we do have up and running in production. Uh DSpark is is new research. Um so this one I can basically only say like hey it exists. It's cool. We're looking at it. Um the difference versus Dlash, it still has that diffusion model, but it also pairs it with a sequential model. And the idea is that we're going to improve acceptance weights um by having these two models work together rather than having just the iterative speculator uh just the diffusion speculator or just the auto reggressive speculator. So DSpark very exciting um but we don't have any production results with it yet to share. Um hopefully you know we'll have those for next time. Um what we do have production results though on is continuous speculator retraining. So this is we're back to Dlash here. Um and this is the idea that you know speculative decoding is very dependent on the actual content of the prompts and responses that you're looking at in your system. And so if you are continuously retraining on those prompts and responses in your live system, you can see a 20% to even 2x improvement in your token acceptance rates. This is actually like really hard to do. Uh it takes a lot of storage and you have to make sure that you have permission to use the data that you're processing in this way. Uh it takes a ton of compute and you have to move all of this information around. And if you change the underlying model, you also have to change the speculator model. But when I look forward into the future, I do think that continuous speculation for very very large scale systems is going to be a worthwhile optimization. So what is next in inference? Um the following is like personal opinion and speculation and public information and like if I knew anything that was actually coming out, I wouldn't be able to talk about it. Um so so this is just like what I think is going to happen. You know I've been through three hardware cycles um through the Ampio release, the Hopper release, the Blackwell release. Um and it always takes time for you know when these chips get shipped to when they get installed in data centers when the entire software stack really is able to take advantage of their capabilities. But some things that I'm excited about are, you know, with with Reuben, it looks like the NVFP4 performance is going to be fantastic. So the more we can like honestly borrow techniques from local inference and get a lot of confidence running models in this NVFP4 data format, the more we're going to be able to take advantage of the awesome performance of of the upcoming Ruben systems. I think that like disagregation and systemwide communication is going to be increasingly important. um we're seeing really excellent early gains from PD disagregation and the ability to you know move information like KV cache data around the system is going to be increasingly important and then like I said the theme of training for inference is going to be something that continues to have a big impact in the industry moving forward so thank you all so much for the talk uh for coming to the talk um I am on Twitter I'm on LinkedIn um and I'm giving out free books um you can download a PDF at the QR code or come down with me to the base 10 booth to get your free copy of Inference Engineering. Uh we've got a bunch there, maybe enough for everyone. If not, we will have a CO bring some more from the office. Um so yeah, I'll be downstairs at the base 10 booth. Thank you all so much and have a great day. \[applause\] \>\> \[music\]