A talk from the AI Engineer World's Fair 2026 on scaling LLM inference to meet rapidly growing token demand. Daniel Kim (Cerebras) covers the runtime layer: why autoregressive decoding is memory-bound, splitting models across GPUs, continuous batching, KV cache growth, paged KV cache, prompt caching, and chunked prefill. Michelle (Gimlet Labs) covers system-level optimization: prefill/decode disaggregation, speculative decoding, attention/FFN disaggregation, Gimlet's multi-silicon inference cloud, and using agents to write optimized kernels. Chapters also touch on the speed-vs-throughput Pareto frontier and memory bandwidth comparisons across hardware. # Video Identity Observations **Video:** *How Inference Keeps Up with Our Insatiable Need for Tokens — Cerebras & Gimlet Labs* ## Intended Audience - **AI infrastructure developers and engineers.** The talk assumes technical fluency, diving into auto-regressive loops, KV cache, prefill/decode phases, and speculative decoding without introductory framing. - **AI builders and practitioners concerned with inference cost/latency.** Examples like *"if you are running a coding agent with very long context, for example, 128k context"* and references to Claude Code fast mode and ChatGPT behavior target people building or operating such products. - **Conference/technical talk attendees.** The speakers repeatedly reference slides (*"you all already know from our previous slides"*) and close with *"Thank you so much for coming to our talk... let us know if you have any questions"* — indicating a live, in-person technical conference audience. ## Claimed Scope - **Inference optimization end-to-end**, from runtime techniques to system-level architecture: - Runtime: batching, continuous batching, paged KV cache, prompt caching, chunked prefill. - System: prefill/decode disaggregation, speculative decoding, attention/FFN disaggregation. - **Future roadmap claims**, including agent-generated kernels and custom silicon: *"we can actually use agents to help us solve these problems and keep pushing the frontier."* - **Industry-scale framing:** opens with spend figures (*"$765 billion on compute and in 2031 up to $1.6 trillion"*) and token demand (*"480 trillion tokens per month"* → *"3.2 quadrillion"*). ## Self-Description Evidence - **Daniel (Cerebras):** *"I work on AI and AI agents here at Cerebras."* Presents Cerebras hardware favorably — *"Cerebras has 7,000 times the bandwidth of a H100"* and *"Cerebras is really built for decode."* - **Michelle (Gimlet Labs):** *"I'm a co-founder at Gimlet Labs where we're building a multi-silicon neo cloud to optimize inference at every layer of the stack."* Later: Gimlet is *"really generalizing that beyond just Trainium and Cerebras"* across *"Nvidia, AMD, Trainium, Cerebras, D-Matrix, Intel Gaudi."* - **Promotional product claims:** Cerebras/AWS partnership promising *"5x high-speed token capacity in the same footprint"*; Gimlet's KForge agent *"able to speed up GPT-OSS-120B on AMD hardware by 33% end-to-end"* and *"three to 10 times faster inference compared to homogeneous systems."* - **Mission framing (creator claim):** *"we are working both very, very hard to give you faster and cheaper tokens"* and *"we're just getting started in our journey to optimize throughput and speed on inference."* ## Distinguishing Claims vs. Demonstrated Patterns - **Creator claims** (unverified here): specific performance numbers (3,000 vs. 600 tokens/sec decode; 10x interactivity), partnership outcomes, and KForge results. - **Demonstrated presentation patterns:** a co-presented vendor talk (two companies) using educational framing of inference fundamentals as a vehicle for promoting each company's products and roadmap; the educational content (Pareto frontier, disagg strategies) is presented as neutral but consistently leads into each speaker's commercial offering. ## Notes for Channel-Level Synthesis - Single conference talk; channel-wide conclusions (cadence, recurring audience, typical format) **cannot be inferred** from this one video. - Tone: technical but playful (*"go eat tokens"*), with humor about marketing language (*"I told Daniel not to say that"* regarding "geared to the right"). # Delivery Observations — Cerebras & Gimlet Labs Inference Talk ## Format & Structure - **Dual-presenter, hand-off style.** Two speakers alternate: Daniel (Cerebras) opens and closes, Michelle (Gimlet Labs) takes the system-level middle section. Explicit hand-offs frame the structure: > "So, now Michelle, tell us more about what else we can optimize." > "So, all of the optimizations that Daniel just talked about are run time level optimizations." - **Linear teaching arc.** The talk follows a deliberate build: problem framing → single-worker optimizations → system-level optimizations → future outlook. Signposting is frequent: > "So in today's talk we're going to talk about two things." > "And as we look towards the future..." - **Problem-then-solution rhythm.** Nearly every technique is introduced by posing a question first: > "So, how can we figure out how to optimize this...?" > "So, how can we solve this problem and squeeze out more performance?" > "So, what are some of the innovations we have done to be able to optimize how we use memory?" ## Recurring Vocabulary - **"So" as a connective habit.** Nearly every paragraph opens or pivots with "So" — "So, what you can do is throw more compute at the problem." "So, the idea here is that..." This is the dominant filler/transition across both speakers. - **Tokens as the unit of everything.** "Insatiable need for tokens" appears twice early; the talk closes on it too: > "we are working both, very, very hard to give you faster and cheaper tokens." > "go eat tokens. We're making more tokens for you..." - **"Compute bound" / "memory bound"** as a repeated diagnostic pair, introduced by Daniel and re-invoked by Michelle: > "This is very compute bound... However, after the first token is generated... this is more reliant on memory." > "prefill, that workload itself is inherently bound by compute. Decode, that is bound by memory." - **"Bound by" and "clogging" metaphors** — pipeline congestion is personified: "it starts to clock up the pipeline... this is kind of catastrophic." - **Casual intensifiers:** "kind of bananas," "juice perf," "optimize the hell out of this," "pretty crazy results," "crazy crazy target." ## Teaching Habits - **Concrete numbers as anchors.** Claims are consistently quantified: "$765 billion on compute," "480 trillion tokens per month... jumped up to 3.2 quadrillion," "42 gigabytes of KV cache," "for every 3,000 tokens, you need 1 GB," "7,000 times the bandwidth of a H100." - **Rule-of-thumb summaries.** After dense explanations, a digestible takeaway is offered: > "So, the rule of thumb here for a 70 billion parameter model is that for every 3,000 tokens, you need 1 GB of additional KV cache." - **Familiar analogies.** Virtual memory explained via "something that's similar to like an OS or a virtual machine"; batching framed as reusing "a weight read that you've already done." - **Rhetorical questions to reset attention:** "And you might be wondering, what is prefill?" "But we can push that even further, right?" - **Tag-questions as verbal tic (Michelle especially):** "And this is going to be memory bound, right?" "just walking through that really quickly again... you're re-reading the entire KB cache, right?" ## Tone & Register - **Conversational-technical blend.** Formal benchmarks sit next to colloquial asides: "AI spend is going kind of bananas right now." - **Self-aware humor, lightly used.** Michelle: "I [laughter] told Daniel not to say that." Closing: "go eat tokens... so you can also generate GIFs like this." A one-off name-drop joke: "it's a recursive loop. Like shout out to the folks at Recursive." - **Vendor enthusiasm without hard selling.** Product claims are embedded in technical argument: "we're really excited to partner with AWS," "So we're really excited to really pursue this partnership." ## Stability Assessment - **Well-supported patterns** (multiple instances): "So"-heavy transitions, token-centric framing, problem→question→solution structure, numeric anchoring, "right?" tag-questions. - **Thin-evidence items** (single occurrences, noted for caution): the humor moments ("told Daniel not to say that," the GIF joke) and the "optimize the hell out of this" register shift — these appear once each, so their consistency across other talks cannot be confirmed from this transcript alone. # Observation Record: "How Inference Keeps Up with Our Insatiable Need for Tokens" — Cerebras & Gimlet Labs ## Speakers - **Daniel** — works on AI and AI agents at Cerebras (self-identified in transcript) - **Michelle** — co-founder at Gimlet Labs, building "a multi-silicon neo cloud to optimize inference at every layer of the stack" (self-identified) --- ## Section 1: Market Scale and Token Demand (Daniel) - AI compute spend figures cited: **$765 billion this year**, projected **up to $1.6 trillion by 2031**. - Frontier labs are described as in a "mad dash" to secure compute across hardware vendors for training and serving. - Google token serving figures as cited by Daniel: **480 trillion tokens/month** (stated at Google IO "last year"), rising to **3.2 quadrillion tokens/month** ("last month"). - Claim that models are getting cheaper and faster: Llama 3 (transcript renders "Lama 3.37 B") is now **25% cheaper and 3.6x faster** than at first release. ## Section 2: Autoregressive Inference and the Compute-Scaling Tax (Daniel) - LLMs generate tokens one at a time in an **autoregressive loop**; each token requires reading all weights from memory into compute. - Claim: computers sit idle waiting on memory **up to 90% of the time** during this process. - **Naive tensor parallelism**: split the model across 4 GPUs to theoretically cut per-token latency 4x — but this carries a communication tax: - An **all-reduce** operation is required after each attention block and each FFN block, at every layer, because each GPU has only partial results. - Stated outcome: only ~**2x speedup for 4x the compute**. - Graphical framing: more GPUs yield faster tokens but require **exponentially more energy per token** due to communication overhead. - Reinterpreted as a **Pareto frontier** between speed (latency) and throughput that inference providers must trade against. - Example cited: **Claude Code fast mode** — reportedly ~6x the tokens per output token vs. regular Claude for ~2x speed (Daniel's framing: "I think, 6x the amount"). - Stated goal of the talk: discuss innovations that **shift the Pareto frontier to the right**. ## Section 3: Runtime-Level Optimizations (Daniel) - **(Continuous) batching**: reuse an already-paid-for weight read to serve multiple requests. Limitation: variable-length requests create idle compute periods. - **Continuous batching**: fill every idle slot with a new request during the same weight read. - Stated downside: KV cache growth. Each generated token writes K/V vectors to the KV cache. - Figures cited: a coding agent with **128k context** requires **42 GB of KV cache** for one request. - Rule of thumb stated: for a **70B parameter model, ~1 GB of KV cache per 3,000 tokens per request**; batching eight such requests would exhaust memory. - **Paged KV cache**: virtual-memory-style provisioning instead of reserved physical blocks, reducing wasted reserved memory. Claimed result: **3x more requests served** vs. the prior approach. - **Prompt caching**: reuse KV cache computed for prior requests (e.g., long system prompts), skipping the prefill phase. - **Prefill vs. decode distinction**: - *Prefill*: request arrival to first token; compute-bound and highly parallelizable. Illustrated by the initial delay before ChatGPT starts responding. - *Decode*: one token at a time reusing KV cache; memory-bound. - **The clogging problem**: extremely long-context coding agents (200K–1M context) block the pipeline — while a large prefill runs, no other prefill or decode requests can proceed. Daniel calls this "kind of catastrophic" in mixed workloads. - **Chunked prefill** as the fix: chop long prefills into smaller pieces and interleave prefill and decode requests, preventing pipeline clogging. ## Section 4: System-Level Optimizations and Disaggregation (Michelle) - Frames all prior content as **runtime-level** (single-worker) optimizations; argues further progress requires **system-level** distribution across many workers. - **Every request differs** in resource profile: - Summarization (long prompt, short answer) → prefill-dominated → **compute-bound**. - Long-form writing (short prompt, long answer) → decode-dominated → **memory-bound**. - Mixed (long/long) → both compute and memory matter. - **Prefill/decode disaggregation**: split workloads into separate prefill and decode worker pools, scaled independently so neither stalls the other. Summarization example: scale up prefill, scale down decode. - Reiterates that decode is memory-bound because each decode step re-reads the entire KV cache. ## Section 5: Hardware Claims and the AWS Partnership (Daniel) - **Memory bandwidth comparison**: H100 = **3.35 TB/s**; Cerebras claims **21,000 TB/s** ("7,000 times the bandwidth"), attributed to a specialized **SRAM-based memory architecture**. - Decode throughput claim: Cerebras serves **3,000 tokens/s** vs. **600 tokens/s** on an H100. - Announced **partnership with AWS Trainium**: Trainium serves prefill; AWS EFA transfers the KV cache to **Cerebras CS-3** systems for decode. Claimed result: **5x high-speed token capacity in the same footprint**. ## Section 6: Multi-Silicon Orchestration (Michelle) - Gimlet generalizes the Trainium/Cerebras pattern into a **multi-silicon inference cloud**: decompose workloads, determine resource characteristics of each piece, map to best-fit hardware (Nvidia, AMD, Trainium, Cerebras, D-Matrix, Intel Gaudi, "the list goes on"). - Claimed result: **3–10x faster inference vs. homogeneous systems with the same power budget**. - **Speculative decoding** (disaggregation by model rather than phase): - A small **draft model** generates **candidate tokens** cheaply; the full model runs a **verify stage** to validate them. - Worked example: 9 tokens via standard decode = 9 full-model forward passes; with speculative decode = **3 full-model passes**, assuming the draft model runs fast enough. - Pareto results presented: baseline is prefill/decode disagg on homogeneous GPUs; speculative decoding on those same GPUs shifts the curve toward speed; running the draft model on fast small-model hardware (D-Matrix, Cerebras) yields claimed **10x interactivity** at the same throughput. - **Attention/FFN disaggregation**: for very large models (~**3 trillion parameters**), split at layer level — attention (many KV cache accesses, memory-bound) on high-bandwidth machines; FFN (dense matrix multiplications) on fast matmul machines — "ping-ponging" across machines within a single model. - Explicit qualification: **"there is no best form of disagg. There is actually no best hardware."** Choice depends on model, workload, latency needs, and throughput guarantees. ## Section 7: Agents as a Future Optimization Loop (Both) - Shared claim: agents are increasingly embedded in work and life, pushing demand for more and faster tokens. (Daniel: "it's really getting geared to the right" — Michelle laughed that she told him not to say that; qualification preserved as expressed.) - **KForge (Gimlet)**: an agent for generating optimized kernels, taking hardware/workload profiles and treating kernel optimization as a brute-force search problem. - Motivation: kernel hand-tuning varies drastically by model, hardware (and hardware generation), and workload shape — intractable across many silicon types. - Result claimed: running less than a week, KForge sped up **GPT-OSS-120B on AMD hardware by 33% end-to-end** — notably "building on top of kernels that have already been optimized." - **Recursive loop thesis** (Michelle): agents help design custom silicon tuned to specific models/workloads → that silicon trains better AI → which designs the next silicon generation. (Acknowledgment: "shout out to the folks at Recursive.") ## Section 8: Closing Optimism (Both) Daniel and Michelle enumerate additional agent-applied optimization surfaces: - **Incident response / uptime**: more reliable systems → more throughput and tokens. - **Model bring-up**: running new models on all hardware types on day one of release. - **Compilers**: just-in-time custom inference software per model/workload/hardware — citing Anthropic as "folks at Anthropic... built a compiler in 2 weeks." - **Tape-out acceleration**: agents speeding custom silicon design. - **Back office**: procurement, legal, and surrounding operational workflows optimized via agentic workflows. Closing framing (Daniel): "we're just getting started"; expressed optimism about serving tokens "faster and cheaper." --- ## Open Tensions and Qualifications Preserved - The **speed-vs-throughput Pareto tradeoff** is presented as fundamental; all innovations discussed are framed as *shifting*, not eliminating, that frontier. - Michelle's explicit disclaimer that no single disaggregation strategy or hardware is universally best. - Daniel's Claude Code pricing figure is hedged ("I think, 6x"); Google and Cerebras performance figures are stated as speaker claims without independent verification within the talk. - Both speakers are commercially interested parties (Cerebras touting its own hardware; Gimlet touting its multi-silicon product); performance claims (3–10x, 5x capacity, 33% kernel speedup) are presented without stated benchmarks methodology in the transcript. # Speaker & Provenance Record ## Video Identification - **Title:** How Inference Keeps Up with Our Insatiable Need for Tokens — Cerebras & Gimlet Labs - **Format:** Conference talk with Q&A invitation; questions directed to speakers after the talk. ## Speakers & Roles | Speaker | Role (self-identified) | Affiliation | |---|---|---| | **Daniel** | AI / AI agents team member | Cerebras | | **Michelle** | Co-founder | Gimlet Labs ("building a multi-silicon neo cloud to optimize inference at every layer of the stack") | - Both speakers present as joint co-presenters; speaking turns alternate throughout the transcript. - No narrator, third-party guest, or other identified speaker appears. - No curated metadata beyond the transcript and title was supplied; speaker identities rest solely on their self-introductions. ## Referenced Sources & Third-Party Claims (as stated by speakers) ### Market / industry figures (attribution as given) - **$765B** compute spend "this year"; up to **$1.6 trillion by 2031** — no source cited in transcript. - **Google I/O** claim: 480 trillion tokens/month served; updated "last month" to **3.2 quadrillion tokens/month**. - **Llama 3.37 B** ("Lama 3.37 B" in transcript): 25% cheaper and 3.6× faster than at initial release — no source cited. ### Third-party products & claims referenced - **Claude Code fast mode:** ~6x token cost for ~2x speed (speaker's approximation: "I think, 6x"). - **Anthropic:** claimed to have "built a compiler in 2 weeks." - **Recursive:** thanked by name ("shout out to the folks at Recursive"). - **H100:** cited memory bandwidth of 3.35 TB/s; **Cerebras** claimed 21,000 TB/s ("7,000 times the bandwidth") and 3,000 tokens/s vs 600 tokens/s decode. ### Partnership announcement (promotional) - **Cerebras + AWS Trainium partnership** to split prefill (Trainium) and decode (Cerebras CS-3) via AWS EFA; claimed **5x high-speed token capacity in the same footprint**. Promotional in nature. ### Company self-promotion (flagged) - **Cerebras:** SRAM-based memory architecture; decode throughput claims; AWS partnership. - **Gimlet Labs:** - Multi-silicon inference cloud across Nvidia, AMD, Trainium, Cerebras, D-Matrix, Intel Gaudi; claimed **3–10x faster inference** vs same-power homogeneous systems. - **KForge** agent for kernel generation: claimed **33% end-to-end speedup of GPT-OSS-120B on AMD hardware** within under a week, building on already-optimized kernels. - The entire talk functions as promotional material for both companies; these claims are vendor-stated and unsourced within the transcript. ## Availability / Interaction - Speakers invite audience questions and state they will "be around after the talk." - No information about comments, transcript distribution, or other availability channels is present. ## Uncertainty Notes - Slides are referenced ("next slide, please") but not available; visual data (Pareto frontier graphs, benchmark tables, KForge example) cannot be verified from the transcript alone. - "Lama 3.37 B" may be a transcription artifact (likely Llama 3.x 70B later referenced); recorded as stated. - All quantitative performance claims are speaker-asserted without cited primary sources. \>\> Hello everyone. My name is Daniel. I work on AI and AI agents here at Cerebras. What's your name? \>\> I'm Michelle. I'm a co-founder at Gimlet Labs where we're building a multi-silicon neo cloud to optimize inference at every layer of the stack. \>\> And we're really excited to bring you our talk, all the things we have to do to satisfy your insatiable need for tokens. As you know, AI spend is going kind of bananas right now. With this year we're slated to spend $765 billion on compute and in 2031 up to $1.6 trillion. The frontier labs are having a mad dash to secure as much compute as possible to buy across hardware vendors to do both training for the frontier models as well as to serve all of their users. And all of their compute is going to satisfy everyone's insatiable need for tokens. Google at their IO last year said that they were serving 480 trillion tokens per month. And last month they said that it jumped up to 3.2 quadrillion tokens per month. So all of this is happening at the same time as models are getting cheaper and faster. Lama 3.37 B when it was first released compared to now is now 25% cheaper whilst still being 3.6 times faster. So in today's talk we're going to talk about two things. One, what have we done so far to optimize inference to make it cheaper and faster. And what does the road map look like to juice perf and throughput even further? So to do this, let's take a little bit of a time jump to 2022 and learn about all the optimizations that led us to the point we're at now and hopefully what the future looks like. So, LLMs produce tokens one at a time in something called an auto-regressive loop. So, to generate one additional token, you need to read all of the weights from memory into compute. So, most of the time, the computer is sitting idle. It's just waiting for memory to be able to stream in all the weights necessary to do the compute itself. So, up to 90% of the time, the computer is just sitting there waiting on memory to load. So, how can we figure out how to optimize this so there's less idle time and you can make full advantage and serve tokens faster? So, what you can do is throw more compute at the problem. So, instead of streaming in the whole set of weights, what if you could split up the entire model into four and have each GPU take care of a fourth of the model each? Then, you could reduce, in theory, the amount of time it takes to serve a single token by four times. However, you have to pay a tax if you want to do this. Before the attention block and then the F So, after the attention block and then the FFM block, you have to do something called all-reduce, where you take each of the individual calculations from each GPU and you combine it all together. So, you can actually get a calculation because each of the GPUs doesn't have the whole picture. It only does 1/4 of the model. So, you need to do this across every layer twice, which can lead to only getting 2x speed up for getting 4x the amount of compute that you add on. If I were to graph this in a pretty graph, um you can see that throwing more GPUs at the problem does get you faster tokens, but it requires exponentially more energy to serve a single token as you put in more compute at the problem because of the communication overhead we just talked about. So, if we flip the Y axis here so it's throughput, you can see that this resembles something like a Pareto frontier, where you can see that you have to make some kind of sacrifice between throughput and speed when you're an inference provider. A great example of this is Claude code fast mode, where you can spend, I think, 6x the amount of tokens to serve a single token compared to regular Claude for twice the speed. So, for that product, they're really optimizing towards the right of the Pareto frontier, where they're getting the speed, but at the cost of not being able to serve as many tokens. So, during today's talk, we're going to talk about what are the innovations we have done in the past and in the future to be able to shift the Pareto frontier to the right. So, you all already know from our previous slides that to generate each token, you have to read the whole model. So, what if we could spend one pass that we already paid for to serve multiple requests? So, that is where the idea of batching and continuous batching comes in. The idea of batching is when you take a weight read that you've already done and be able to serve multiple user requests. However, the problem, though, is that some of these requests are longer than others, which means that there's all of these idle periods in the compute where nothing is happening. So, how can we solve this problem and squeeze out more performance? That's where continuous batching comes in. So, the idea here is that for every single idle slot, you slot in a new user request to be served during that same weight read to be able to serve as many requests as possible. However, this does come with a slight downside because every single token when you do inference requires you to save something into the KV cache because that's how the attention block works. You need to understand how every single token, like what the K and V vectors are. This can add up very quickly. So, if you are running a coding agent with very long context, for example, 128k context, you're going to save 42 gigabytes of KV cache that is needed to serve that individual request. So, the rule of thumb here for a 70 billion parameter model is that for every 3,000 tokens, you need 1 GB of additional KV cache. And that's for one user request. So, imagine you're doing batching and you're doing like eight different requests at the same time, then your KV cache is going to explode and you will not have enough memory to be able to serve all of your requests. So, what are some of the innovations we have done to be able to optimize how we use memory? So, that's where paged KV cache comes into play. So, how traditionally KV cache worked was that you would have physical memory that you would reserve to make sure that you could have enough room to be able to store all of the KV information from each request into memory. However, this came with a downside and a lot of inefficiency because you would just have a lot of memory blocks that would be reserved but wouldn't actually be used. So, the idea with a paged KV cache is we use something that's similar to like an OS or a virtual machine where you provision a whole memory block that's all virtual and you can store wherever there's free space KV cache for each request. So, this allows you to serve three times more requests than not having a paged KV cache and just doing what was before. And this really reduces the amount of wasted space where you are reserving memory slots without actually using them. This was right after followed by an innovation called prompt caching. So, the idea here is that if you have a coding agent or if you have a chat app that has a very long system prompt, you don't actually have to recalculate the KV cache needed every single time you have a request. What you can do is you can reuse the KV cache that you calculated already for the previous request for the next request. That way you can just pick up where you left off. Here, you can skip the very, very time-consuming prefill phase. And you might be wondering, what is prefill? So, during inference, you can split up what happens into two major phases, prefill and decode. So, the prefill phase consists of all the things that happen once the request arrives, and then ends when the first tokens gets generated. And here, you can see this when you type something into ChatGPT, and then it takes a couple seconds before it starts returning the first token, and then it goes kind of like smoothly from there. So, what's happening during the first couple of seconds where it's not returning anything is that it's computing K and V vectors for every single prompt token that you've inputted and produces the first output token. This is very compute bound because you can optimize the hell out of this. You can parallelize this extremely. However, after the first token is generated, what happens is uh that you decode one token at a time, reusing the KV cache you just built in the prefill phase. So, this is more reliant on memory. The reason I'm talking about this is because something happened lately that's really messing up a lot of inference pipelines, which is super super token hungry coding agents. So, let's say you have a coding agent that's using 200K context or 500K context or a million context, right? What happens is that it starts to clock up the pipeline. While prefill is running, no other prefill or decode request can get through. So, this is kind of catastrophic if you have a bunch of people that are sending in coding requests and a couple people that are just trying to figure out like math problems in chat mode. So, to avoid this clogging problem, there is an innovation called chunked prefill that kind of goes and solves this problem. And the idea is quite simple here. The idea is taking a very long piece of prefill, chopping it up into smaller bits, so you can interleave in different decode and prefill requests. That way, there's not one thing that's taking up a bunch of time and clogging the whole pipeline. So, now Michelle, tell us more about what else we can optimize. \>\> Yeah, so all of the optimizations that Daniel just talked about are run time level optimizations. So, that explains we explore how we make inference more efficient within a single worker. But, in order to to keep pushing that frontier, we really need to start looking at system level optimizations, which is how can we distribute these workloads across many different workers. And so, the first thing to recognize there is that every request is different. So, take for example a summarization use case. You're throwing the LLM a bunch of different documents, and you just really want a single question answered. And so, this workload is going to have a long prompt and short answer. And so, it's a dominated by that prefill phase, and so therefore compute bound. On the other hand, you might have a case where it's the opposite. Let's say you want to write a long story or book, you just want to pass it a few themes for it to go off of. And so, this is going to have a short prompt and long answer. And this is going to be dominated by decode because you have so many output tokens compared to the number of input tokens coming in. And this is going to be memory bound. And finally, you can have a case that's somewhere in the middle, right? It's both a long prompt, long answer, and you're seeing here that both the prefill and decode phases and therefore compute and memory both matter. And by looking at these different examples, you can really see that the resource requirements that you have for your hardware, it's pretty different depending on what your workload looks like. And so, how do you prevent yourself from getting, you know, bound by compute, bound by memory? And so, one way to do that is called disaggregation. So, splitting up your workload to different worker nodes. And so, the most common form is called prefill decode disagg, where you're actually splitting and disaggregating this workload by phase. And what that allows you to do is essentially have two pools. So, you have your prefill pool, that's going to be running the prefill part of your workload. The decode pool, that's going to be running the decode part of your workload. And taking the summarization use case for example, you know, you're going to want to scale up that prefill pool so you're not blocked by the compute and scale down your decode pool because you don't want to waste resources when you just really don't have that much decode work to do. And it's nice cuz now you can scale all these independently and neither part is stalling the other. And we can take that one step further, right? We said before, prefill, that workload itself is inherently bound by compute. Decode, that is bound by memory. And just walking through that really quickly again, the reason for that is as you're doing decode, you're stepping through each of these tokens one by one and for every decode step, you're re-reading the entire KB cache, right? That's going to be memory bound. \>\> So yeah, um the the thing that Michelle just said about this being memory bound, this can be clearly seen by what the hardware can actually do. So GPUs, right? Um the the speed at which you can transfer data from memory into compute is limited by something called memory bandwidth. So for an H100, this looks like 3.35 terabytes per second. However, Cerebras has 7,000 times the bandwidth of a H100 with 21,000 terabytes per second. This is because we have a very specialized SRAM based memory architecture on our chip. So this really means that Cerebras is really built for decode. So we can serve tokens at 3,000 tokens per second versus an H100 serving it at 600 tokens per second for decode. So we're really excited to partner with AWS to productionize this. So we are going to partner with AWS Trainium so they can serve prefill using their AWS Trainium chips, use their EFA service to transfer the KB cache to our Cerebras CS-3 systems to finally serve the the token. So this will lead us to give us 5x high-speed token capacity in the same footprint. So, this is one of the things that we're doing to try to make fast inference more accessible and cheaper for developers. So, we're really excited to really pursue this partnership. \>\> And at Gimlet, we're really generalizing that beyond just training them and Cerebras. What Gimlet is building is essentially a multi-silicon inference cloud, which takes your workload, breaks it down into finer-grain pieces, determines the resource characteristics of all of those different pieces, and then figures out the best way to map it to the underlying hardware, where that hardware can be really anything, right? Nvidia, AMD, Trainium, Cerebras, D-Matrix, Intel Gaudi. The list goes on and on. And by running those tasks on the machine that is most efficient for running it, we're actually seeing pretty crazy results. So, we're seeing three to 10 times faster inference compared to homogeneous systems that have the same power budget. And this is diving into more than just the pre-full decode disaggregation than before, and some of those techniques are, for example, disaggregating based on model instead of on phase. And so, that's called speculative decoding. So, just reviewing again the decode phase, typically you do a forward pass for each of the decode tokens that you're getting out. And that's the standard way of doing it. What speculative decode is is that you essentially have a draft model. And this is just a smaller form of the full model that um can generally approximate the results that you get out of it. And the nice thing there is because it's smaller, it runs a lot more quickly and also uses less resources. And so, in speculative decode, what you can do is, in order to get, you know, the nine next tokens, you will essentially run the decode on that draft model first. It's going to give you back what is called candidate tokens. So, you're not 100% sure all these tokens are right, but because, you know, often these have been fine-tuned or been trained off of the full model, it's it's approximately right. And you want to validate, okay, are these the tokens that I actually want to output? And so you'll actually run the next phase on that full model that's called the verify stage to validate, okay, which of these tokens are correct? And essentially you can keep doing this until, you know, you've validated all these tokens from the draft model. And at the end of the day you can see here on the top, the standard way of running it, you needed to run that forward pass on your full model nine times to get those nine tokens. On the other hand, with speculative decode, you actually only needed to run it three times on that full model. And if you can get that draft model to run really, really quickly, then that time is minimal and you can really shorten the amount of time it takes to get your tokens. And looking at some more concrete results, so this is essentially that Pareto frontier that Daniel was showing earlier where x-axis is the speed at which you're getting the tokens and the y-axis is the throughput. And so the baseline case we're comparing to here is prefill decode disagg across homogeneous GPUs. And you can see that, you know, throughput is pretty good, but as soon as you're pushing on that speed, that throughput drops pretty drastically. So what if you then run speculative decode across those same GPUs? So now you're running that draft model and you're able to see you're able to get that curve to the right. And the reason for that is because you're running the decode on that full model less times. But we can push that even further, right? Like going back to our theme of running those workloads on the hardware that can run it best. On something such as, you know, D Matrix or Cerebrus where you're able to run small models very, very quickly, you see drastic performance improvements. So here this is where you're seeing those numbers for that same throughput, right? You're able to get 10x the interactivity compared to if you're just running it on homogeneous machines. The final form of disagg we wanted to talk about is called attention FFN disagg. And this one gets pretty exciting when you think about serving large models. And I'm saying like 3 trillion parameter models where these models are so large that even just running one layer, it can be pretty resource-intensive. And so the concept is essentially the same thing, right? Like for that particular task that you need to do, whether it's that attention layer or the feedforward FFN network layer, then you are essentially trying to figure out, okay, what are the resource characteristics, then what is the best hardware to map it to? And so the attention layer, that's going to be more memory bound. It's doing a lot of KB cache accesses. The FFN layer, that's doing a lot of dense matrix multiplications. And so then it makes sense, right? You're going to run the attention on the machine that has memory bandwidth, the FFN layer on the machines that can run those matrix multiplications as quickly as possible. And essentially what happens here is even within the model, you're ping-ponging across, you know, many different machines in order to to get the inference from that model and just really disaggregating every single layer. And one thing that we really wanted to note here is that there is no best form of disagg. There is actually no best hardware. It really depends on the tradeoffs that you have for, you know, depending on what your model is, your workload, and what you need for latency, and what you need for throughput guarantees. And as we look towards the future, we see that agents are only becoming more and more incorporated into our everyday life. It's incorporated into our work. Everyone is using it. And it's continuing to push that need for more and faster tokens. \>\> Yeah, it's really getting geared to the right. \>\> I \[laughter\] told Daniel not to say that. But the good thing about this is that even though, you know, this seems like a crazy crazy target, we can actually use agents to help us solve these problems and keep pushing the frontier. And so one of the things that we're doing at Gimlet is actually using an agent to generate optimized kernels. Especially for our use case where we want to support such a wide range of different kinds of hardware, we need to figure out how do you make that problem a lot more tractable. Cuz right now, how people generate and how people optimize kernels is that there's a lot of people sitting there hand tuning and figuring out how to optimize these kernels. And the thing is that these optimizations can vary pretty drastically based on you know, the model that you're trying to run, the hardware that you're running on, even within generations of the hardware, and you know, the workload shape itself. And so what we do at Gimlet is we built KForge, which is an agent for generating these kernels. It takes in a bunch of information like your hardware profiles, your workload profiles, basically anything that you can use to figure out how to optimize kernels and we solve it as a brute force problem, right? And so here's an example, right? We have a forward pass and the kernel that um KForge optimized from it. And we're seeing pretty interesting results. So last week, you know, we were running the agent for probably less than the span of a week and within that time it was able to speed up GPT-OSS-120B on AMD hardware by 33% end-to-end. And the interesting thing to note here is that this is building on top of kernels that have already been optimized. And you can go and take, you know, agents to the next level and use them to help us build and design custom silicon that is hyper tuned for specific models or your specific workload. This can then be used to train even better and smarter and faster AI, which then can also be used then to design the next generation of that custom silicon. And you know, this loop keeps going on and on and it's a recursive loop. Like shout out to the folks at Recursive. \>\> And I want to end the talk by saying that we're just getting started in our journey to optimize throughput and speed on inference. Because at the very top of the stack, you don't think really about inference optimization when you talk about like incident response, right? But when you have agents that are making our systems much more reliable and increasing uptime, we're getting more throughput and we can get more tokens. \>\> Or you can apply agents to the problem of bringing up models. As soon as a model drops on the market, getting it running on all the different kinds of hardware on day one. \>\> You can even go a little bit crazier with compilers. Imagine a world where an inference that can be custom-built for your workload and the model that you're running. This is already being demonstrated by folks at Anthropic, right? That built a compiler in 2 weeks. So, I think there's a lot of optimizations that you can do when you're building just-in-time custom software to support the model you want to run, on the workload you want to run, on your custom hardware. \>\> And referring back to the previous slide, using agents to continue to help us to build, you know, more custom silicon and helping speed up the tape-out process. \>\> And finally, this is the non-very technical part, but all of the things that surround what takes to serve inference. All the procurement, all the legal, all of the things that are in the back office. All of these things combined with agentic workflows are able to be optimized. So, when we do all of this, I'm really optimistic about the future and all the things we can accomplish to serve tokens faster and cheaper. So, next slide, please. Thank you so much for coming to our talk. Uh we are working both very, very hard to give you faster and cheaper tokens. And yeah, let us know if you have any questions. We'll be around after the talk. \>\> Yeah, and go eat tokens. We're making more tokens for you so you can also generate GIFs like this. \>\> Yeah. \>\> \[laughter\]