--- schema: 1 type: video title: 'What Makes Open Models Fast in Production — Sujee Maniyam, Nebius' description: >- Open models have nearly caught up with proprietary ones. Serving them fast and cheaply is the hard part. Dylan Bristot, who leads product marketing for Nebius Token Factory, and developer advocate Sujee Maniyam explain what it takes to run open LLMs in production. Dylan covers the trade-off between closed APIs and self-hosting, and the inference → data → post-training → deployment loop. Sujee then goes layer by layer through the optimizations behind fast inference: NVFP4 on the latest NVIDIA hardware, engine selection, cache-aware routing, speculative decoding with custom draft models, KV cache offloading, disaggregated prefill and decode, and finding the quantization sweet spot. In this talk: • Why open models are now competitive, and what that means for cost and lock-in • Cache-aware routing versus naive load balancing for LLMs • Speculative decoding and training draft models on your own traffic • KV cache offloading, prefill/decode disaggregation and quantization trade-offs SPEAKERS Sujee Maniyam, AI Developer Advocate, Nebius LinkedIn: https://www.linkedin.com/in/sujeemaniyam/ X: https://x.com/sujee_dev Website: https://sujee.dev/ Dylan Bristot, Senior AI Product Marketing Manager, Nebius LinkedIn: https://www.linkedin.com/in/dylanbristot/ LINKS Nebius Token Factory: https://tokenfactory.nebius.com/ Token Factory docs: https://docs.tokenfactory.nebius.com/quickstart Nebius: https://nebius.com/ CHAPTERS 0:00 Intro 0:12 Engineering open LLMs for production 0:52 Who is Nebius 2:27 Nebius Token Factory 2:42 Closed APIs vs self-hosting 4:06 The full loop: inference, data, post-training, deploy 6:21 What we handle so you don't have to 8:06 From running models to improving them 10:01 Scaling open models to production 10:26 How good are open models? 11:26 Teacher and student models 11:46 Hardware layer: NVFP4 12:26 Choosing the serving engine 12:55 Cache-aware routing 14:24 Speculative decoding 15:29 Training custom draft models 15:59 KV cache and offloading 17:33 Disaggregated prefill and decode 18:13 Finding the quantization sweet spot 18:43 Recap 19:22 Try Token Factory Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMInference #OpenSourceAI #AIEngineer canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/what-makes-open-models-fast-in-production-sujee-man-59df3f5b1428 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/what-makes-open-models-fast-in-production-sujee-man-59df3f5b1428.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:20:04.647Z' language: null children: [] videoId: TRe1u7dHYiA durationSeconds: 1232 publishedAt: '2026-10-03T22:30:28.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 168 endSeconds: 179 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=168' heldReason: rights - ordinal: 1 text: null startSeconds: 179 endSeconds: 254 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=179' heldReason: rights - ordinal: 2 text: null startSeconds: 254 endSeconds: 264 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=254' heldReason: rights - ordinal: 3 text: null startSeconds: 264 endSeconds: 384 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=264' heldReason: rights - ordinal: 4 text: null startSeconds: 395 endSeconds: 494 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=395' heldReason: rights - ordinal: 5 text: null startSeconds: 505 endSeconds: 601 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=505' heldReason: rights - ordinal: 6 text: null startSeconds: 635 endSeconds: 645 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=635' heldReason: rights - ordinal: 7 text: null startSeconds: 688 endSeconds: 699 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=688' heldReason: rights - ordinal: 8 text: null startSeconds: 722 endSeconds: 746 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=722' heldReason: rights - ordinal: 9 text: null startSeconds: 746 endSeconds: 757 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=746' heldReason: rights - ordinal: 10 text: null startSeconds: 788 endSeconds: 865 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=788' heldReason: rights - ordinal: 11 text: null startSeconds: 865 endSeconds: 877 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=865' heldReason: rights - ordinal: 12 text: null startSeconds: 933 endSeconds: 944 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=933' heldReason: rights - ordinal: 13 text: null startSeconds: 965 endSeconds: 975 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=965' heldReason: rights - ordinal: 14 text: null startSeconds: 975 endSeconds: 1062 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=975' heldReason: rights - ordinal: 15 text: null startSeconds: 1074 endSeconds: 1095 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=1074' heldReason: rights - ordinal: 16 text: null startSeconds: 1095 endSeconds: 1105 watch: 'https://liquidgold.cc/watch/TRe1u7dHYiA?t=1095' heldReason: rights --- # What Makes Open Models Fast in Production — Sujee Maniyam, Nebius Open models have nearly caught up with proprietary ones. Serving them fast and cheaply is the hard part. Dylan Bristot, who leads product marketing for Nebius Token Factory, and developer advocate Sujee Maniyam explain what it takes to run open LLMs in production. Dylan covers the trade-off between closed APIs and self-hosting, and the inference → data → post-training → deployment loop. Sujee then goes layer by layer through the optimizations behind fast inference: NVFP4 on the latest NVIDIA hardware, engine selection, cache-aware routing, speculative decoding with custom draft models, KV cache offloading, disaggregated prefill and decode, and finding the quantization sweet spot. In this talk: • Why open models are now competitive, and what that means for cost and lock-in • Cache-aware routing versus naive load balancing for LLMs • Speculative decoding and training draft models on your own traffic • KV cache offloading, prefill/decode disaggregation and quantization trade-offs SPEAKERS Sujee Maniyam, AI Developer Advocate, Nebius LinkedIn: https://www.linkedin.com/in/sujeemaniyam/ X: https://x.com/sujee_dev Website: https://sujee.dev/ Dylan Bristot, Senior AI Product Marketing Manager, Nebius LinkedIn: https://www.linkedin.com/in/dylanbristot/ LINKS Nebius Token Factory: https://tokenfactory.nebius.com/ Token Factory docs: https://docs.tokenfactory.nebius.com/quickstart Nebius: https://nebius.com/ CHAPTERS 0:00 Intro 0:12 Engineering open LLMs for production 0:52 Who is Nebius 2:27 Nebius Token Factory 2:42 Closed APIs vs self-hosting 4:06 The full loop: inference, data, post-training, deploy 6:21 What we handle so you don't have to 8:06 From running models to improving them 10:01 Scaling open models to production 10:26 How good are open models? 11:26 Teacher and student models 11:46 Hardware layer: NVFP4 12:26 Choosing the serving engine 12:55 Cache-aware routing 14:24 Speculative decoding 15:29 Training custom draft models 15:59 KV cache and offloading 17:33 Disaggregated prefill and decode 18:13 Finding the quantization sweet spot 18:43 Recap 19:22 Try Token Factory Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMInference #OpenSourceAI #AIEngineer Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/what-makes-open-models-fast-in-production-sujee-man-59df3f5b1428 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.