--- schema: 1 type: video title: >- Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta description: >- Inference traffic at Meta already outpaces the largest microservices in the world, the fastest growing workload it has ever run. Nishant Gupta's frame for what comes next is 2008. The cloud began as virtual machines, then value climbed to schedulers, service meshes and autoscalers, until orchestration held the complexity and the value. AI is on the same path, compressed into a few years, with orchestration emerging around routing, KV cache management and prefill decode disaggregation. His sharpest point is coupling. None of the layers is new. What is new: a routing decision changes the cache hit rate, which changes batch composition, which changes GPU utilization, which changes the autoscaler's next move. A request is now a distributed transaction where every hop can retry, time out or fail; preempt a GPU after 200 tokens have streamed and you cannot simply retry. Reliability has to live in the control plane, the only layer that sees the whole workflow. A scheduler for inference has to see seven axes: GPU generation, memory headroom, KV cache state, whether weights are warm, tenant priority, latency budget, and workflow context, since failing step three of five wastes what steps one and two already spent. Every optimization fits one of four quadrants: avoid the work through caching, share it through batching, move it through routing to a cheaper model or region, or delay it through admission control. The metric to optimize is cost per successful task, not per token. Naman Ahuja takes over: observability as a control loop's input, the latency, cost and throughput triangle where every serving decision moves you somewhere, and the conclusion that inference needs its own control plane, the way virtual machines needed Kubernetes. Speaker info: - https://www.linkedin.com/in/nishantgupta-ai/ - https://buzzingtech.ai/ - https://www.linkedin.com/in/namanahuja/ Timestamps: 0:00 - Inference is the fastest growing workload Meta has seen 0:54 - 2008 again: value moves to the orchestration layer 3:13 - Microservices versus inference, dimension by dimension 6:09 - Old layers, new coupling 7:36 - A request is a distributed transaction 8:31 - Seven axes a scheduler has to see 10:19 - Avoid, share, move, or delay the work 12:36 - Cascading failure, with a KV cache twist 16:19 - The latency, cost, throughput triangle 17:00 - Inference needs its own control plane --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/7c9FSUVcXR0-youtube-41ce07126fda01961f0f Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/7c9FSUVcXR0' markdown: 'https://liquidgold.cc/watch/7c9FSUVcXR0.md' llms: 'https://liquidgold.cc/watch/7c9FSUVcXR0/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:21:07.379Z' language: null children: [] videoId: 7c9FSUVcXR0 durationSeconds: 1191 publishedAt: '2026-09-19T15:00:10.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 23 endSeconds: 57 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=23' heldReason: rights - ordinal: 1 text: null startSeconds: 67 endSeconds: 193 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=67' heldReason: rights - ordinal: 2 text: null startSeconds: 67 endSeconds: 193 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=67' heldReason: rights - ordinal: 3 text: null startSeconds: 204 endSeconds: 371 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=204' heldReason: rights - ordinal: 4 text: null startSeconds: 204 endSeconds: 371 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=204' heldReason: rights - ordinal: 5 text: null startSeconds: 204 endSeconds: 371 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=204' heldReason: rights - ordinal: 6 text: null startSeconds: 204 endSeconds: 371 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=204' heldReason: rights - ordinal: 7 text: null startSeconds: 371 endSeconds: 384 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=371' heldReason: rights - ordinal: 8 text: null startSeconds: 459 endSeconds: 472 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=459' heldReason: rights - ordinal: 9 text: null startSeconds: 472 endSeconds: 516 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=472' heldReason: rights - ordinal: 10 text: null startSeconds: 526 endSeconds: 625 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=526' heldReason: rights - ordinal: 11 text: null startSeconds: 625 endSeconds: 636 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=625' heldReason: rights - ordinal: 12 text: null startSeconds: 636 endSeconds: 763 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=636' heldReason: rights - ordinal: 13 text: null startSeconds: 636 endSeconds: 763 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=636' heldReason: rights - ordinal: 14 text: null startSeconds: 763 endSeconds: 774 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=763' heldReason: rights - ordinal: 15 text: null startSeconds: 763 endSeconds: 774 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=763' heldReason: rights - ordinal: 16 text: null startSeconds: 774 endSeconds: 983 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=774' heldReason: rights - ordinal: 17 text: null startSeconds: 983 endSeconds: 995 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=983' heldReason: rights - ordinal: 18 text: null startSeconds: 995 endSeconds: 1026 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=995' heldReason: rights - ordinal: 19 text: null startSeconds: 1026 endSeconds: 1038 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=1026' heldReason: rights - ordinal: 20 text: null startSeconds: 1038 endSeconds: 1191 watch: 'https://liquidgold.cc/watch/7c9FSUVcXR0?t=1038' heldReason: rights --- # Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta Inference traffic at Meta already outpaces the largest microservices in the world, the fastest growing workload it has ever run. Nishant Gupta's frame for what comes next is 2008. The cloud began as virtual machines, then value climbed to schedulers, service meshes and autoscalers, until orchestration held the complexity and the value. AI is on the same path, compressed into a few years, with orchestration emerging around routing, KV cache management and prefill decode disaggregation. His sharpest point is coupling. None of the layers is new. What is new: a routing decision changes the cache hit rate, which changes batch composition, which changes GPU utilization, which changes the autoscaler's next move. A request is now a distributed transaction where every hop can retry, time out or fail; preempt a GPU after 200 tokens have streamed and you cannot simply retry. Reliability has to live in the control plane, the only layer that sees the whole workflow. A scheduler for inference has to see seven axes: GPU generation, memory headroom, KV cache state, whether weights are warm, tenant priority, latency budget, and workflow context, since failing step three of five wastes what steps one and two already spent. Every optimization fits one of four quadrants: avoid the work through caching, share it through batching, move it through routing to a cheaper model or region, or delay it through admission control. The metric to optimize is cost per successful task, not per token. Naman Ahuja takes over: observability as a control loop's input, the latency, cost and throughput triangle where every serving decision moves you somewhere, and the conclusion that inference needs its own control plane, the way virtual machines needed Kubernetes. Speaker info: - https://www.linkedin.com/in/nishantgupta-ai/ - https://buzzingtech.ai/ - https://www.linkedin.com/in/namanahuja/ Timestamps: 0:00 - Inference is the fastest growing workload Meta has seen 0:54 - 2008 again: value moves to the orchestration layer 3:13 - Microservices versus inference, dimension by dimension 6:09 - Old layers, new coupling 7:36 - A request is a distributed transaction 8:31 - Seven axes a scheduler has to see 10:19 - Avoid, share, move, or delay the work 12:36 - Cascading failure, with a KV cache twist 16:19 - The latency, cost, throughput triangle 17:00 - Inference needs its own control plane --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/7c9FSUVcXR0-youtube-41ce07126fda01961f0f Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/7c9FSUVcXR0 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.