--- schema: 1 type: video title: Is Speculative Decoding Worth It? Profiling vLLM on NVIDIA Blackwell — Akamai description: >- A small model guesses the next few tokens. The big model checks them in one pass. When is that worth it? Sheilah Kirui, developer advocate at Akamai, explains speculative decoding and how to tell whether it's worth turning on. She walks through the prefill and decode phases of inference, how a small draft model proposes tokens that the target model verifies in a single forward pass, and the memory cost of hosting a second model and its KV cache. She covers how to choose a draft model, then demos both setups side by side with vLLM on a single NVIDIA Blackwell GPU: structured output ran 1.6x faster with a high acceptance rate, while creative writing saw far lower acceptance. She also explains why long-context and high-concurrency workloads gain less. In this talk: • Prefill vs decode, and what speculative decoding speeds up • Choosing a draft model: size, shared tokenizer, cost and accuracy • Demo: acceptance rate on structured vs creative tasks in vLLM • When it's worth it: spare VRAM, small batches, structured output, short context SPEAKER Sheilah Kirui, Developer Advocate, Akamai LINKS Akamai developers on GitHub: https://github.com/akamai-developers vLLM docs: https://docs.vllm.ai Akamai: https://www.akamai.com Akamai on X: https://x.com/Akamai CHAPTERS 0:00 Intro 0:37 Prefill and decode 1:37 The idea behind speculative decoding 2:42 The cost: two models in memory 3:26 Choosing a draft model 3:56 Demo setup on one Blackwell GPU 5:41 Which workloads benefit 6:21 Demo: structured output 9:00 Demo: 1.6x faster 9:50 Demo: creative writing 10:55 Long context limits the gains 11:50 Checklist 12:39 Q&A: tools and resources Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMInference #vLLM #AIEngineer canonical: 'https://liquidgold.cc/watch/XTpyNrEgJQ4' markdown: 'https://liquidgold.cc/watch/XTpyNrEgJQ4.md' llms: 'https://liquidgold.cc/watch/XTpyNrEgJQ4/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:19:43.473Z' language: null children: [] videoId: XTpyNrEgJQ4 durationSeconds: 917 publishedAt: '2026-10-06T22:30:02.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 58 endSeconds: 99 watch: 'https://liquidgold.cc/watch/XTpyNrEgJQ4?t=58' heldReason: rights - ordinal: 1 text: null startSeconds: 99 endSeconds: 110 watch: 'https://liquidgold.cc/watch/XTpyNrEgJQ4?t=99' heldReason: rights - ordinal: 2 text: null startSeconds: 110 endSeconds: 167 watch: 'https://liquidgold.cc/watch/XTpyNrEgJQ4?t=110' heldReason: rights - ordinal: 3 text: null startSeconds: 167 endSeconds: 178 watch: 'https://liquidgold.cc/watch/XTpyNrEgJQ4?t=167' heldReason: rights - ordinal: 4 text: null startSeconds: 178 endSeconds: 214 watch: 'https://liquidgold.cc/watch/XTpyNrEgJQ4?t=178' heldReason: rights - ordinal: 5 text: null startSeconds: 214 endSeconds: 226 watch: 'https://liquidgold.cc/watch/XTpyNrEgJQ4?t=214' heldReason: rights - ordinal: 6 text: null startSeconds: 250 endSeconds: 347 watch: 'https://liquidgold.cc/watch/XTpyNrEgJQ4?t=250' heldReason: rights - ordinal: 7 text: null startSeconds: 357 endSeconds: 381 watch: 'https://liquidgold.cc/watch/XTpyNrEgJQ4?t=357' heldReason: rights - ordinal: 8 text: null startSeconds: 557 endSeconds: 595 watch: 'https://liquidgold.cc/watch/XTpyNrEgJQ4?t=557' heldReason: rights - ordinal: 9 text: null startSeconds: 609 endSeconds: 667 watch: 'https://liquidgold.cc/watch/XTpyNrEgJQ4?t=609' heldReason: rights - ordinal: 10 text: null startSeconds: 667 endSeconds: 677 watch: 'https://liquidgold.cc/watch/XTpyNrEgJQ4?t=667' heldReason: rights - ordinal: 11 text: null startSeconds: 712 endSeconds: 727 watch: 'https://liquidgold.cc/watch/XTpyNrEgJQ4?t=712' heldReason: rights - ordinal: 12 text: null startSeconds: 768 endSeconds: 779 watch: 'https://liquidgold.cc/watch/XTpyNrEgJQ4?t=768' heldReason: rights --- # Is Speculative Decoding Worth It? Profiling vLLM on NVIDIA Blackwell — Akamai A small model guesses the next few tokens. The big model checks them in one pass. When is that worth it? Sheilah Kirui, developer advocate at Akamai, explains speculative decoding and how to tell whether it's worth turning on. She walks through the prefill and decode phases of inference, how a small draft model proposes tokens that the target model verifies in a single forward pass, and the memory cost of hosting a second model and its KV cache. She covers how to choose a draft model, then demos both setups side by side with vLLM on a single NVIDIA Blackwell GPU: structured output ran 1.6x faster with a high acceptance rate, while creative writing saw far lower acceptance. She also explains why long-context and high-concurrency workloads gain less. In this talk: • Prefill vs decode, and what speculative decoding speeds up • Choosing a draft model: size, shared tokenizer, cost and accuracy • Demo: acceptance rate on structured vs creative tasks in vLLM • When it's worth it: spare VRAM, small batches, structured output, short context SPEAKER Sheilah Kirui, Developer Advocate, Akamai LINKS Akamai developers on GitHub: https://github.com/akamai-developers vLLM docs: https://docs.vllm.ai Akamai: https://www.akamai.com Akamai on X: https://x.com/Akamai CHAPTERS 0:00 Intro 0:37 Prefill and decode 1:37 The idea behind speculative decoding 2:42 The cost: two models in memory 3:26 Choosing a draft model 3:56 Demo setup on one Blackwell GPU 5:41 Which workloads benefit 6:21 Demo: structured output 9:00 Demo: 1.6x faster 9:50 Demo: creative writing 10:55 Long context limits the gains 11:50 Checklist 12:39 Q\&A: tools and resources Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMInference #vLLM #AIEngineer Canonical: https://liquidgold.cc/watch/XTpyNrEgJQ4 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:12 to 1:39) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (1:39 to 2:47) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (2:47 to 3:34) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (3:34 to 3:58) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (3:58 to 5:47) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (5:47 to 6:21) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (6:21 to 9:05) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (9:05 to 9:55) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 8: Document excerpt (9:55 to 11:07) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 9: Document excerpt (11:07 to 11:52) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 10: Document excerpt (11:52 to 12:48) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 11: Document excerpt (12:48 to 15:17) > Held: transcript rights do not permit this chunk's text.