Talk by Nishant Gupta and Naman Ahuja (Meta) on operating distributed AI inference systems at scale. The talk argues that AI inference is following the same trajectory as cloud computing after 2008, with value migrating from individual layers to orchestration. Key topics: inference as Meta's fastest-growing workload; why a request behaves like a distributed transaction (coupling between routing, KV cache hit rates, batching, GPU utilization, and autoscaling); reliability in the control plane; the seven axes an inference scheduler must consider (GPU generation, memory headroom, KV cache state, weight warmth, tenant priority, latency budget, workflow context); four optimization quadrants (avoid, share, move, delay); optimizing cost per successful task rather than per token; and the latency/cost/throughput tradeoff triangle, concluding that inference needs its own dedicated control plane. ## Video Identity Observation ### Source Overview - **Title:** "Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta" - **Speakers:** Self-identify in the transcript: *"My name is Nishan Gupta and I'm joined today by my co-speaker Naman Auja. We work on building the efficiency training and inference infrastructure at Meta."* - **Format evidence:** Conference talk — *"the first infants talk on the last day of AI engineer world fair"* — indicating this is a recorded presentation from the **AI Engineer World's Fair**, not native channel content. ### Intended Audience The talk addresses **infrastructure and platform engineers**, explicitly assuming technical depth: - *"most of us in this room have a deep expertise across one or two or three layers. We might own kernels or kernel optimizations... routing or the product itself... operating the GPU infrastructure or the cluster itself."* - Familiarity assumed with Kubernetes, vLLM/SGLang/TensorRT, KV cache, prefill/decode disaggregation, continuous batching, and SLOs (TTFT, time-between-tokens) — none are explained from scratch. ### Claimed Scope - **Topic scope:** Operation of distributed LLM inference at hyperscale, focused on orchestration rather than modeling: *"It's not just about the best models or the kernels or the optimizations. It's about the whole ecosystem. It's about the control plane and the orchestration."* - **Industry-trend claim:** AI infrastructure follows a compressed cloud-era trajectory — *"AI is on the exact same trajectory but just compressed into a last few years instead of a decade."* - **Central thesis the channel/talk advances:** *"Infrastructure is no longer a server problem. It's an orchestration problem"* and *"The teams that understand this early on will build infrastructure for the future."* ### Self-Description Evidence - **Affiliation claim:** Meta infrastructure team (efficiency training and inference infrastructure), per the opening lines. - **No channel-level self-description exists** — no statements about the publishing channel, its series, or its broader content strategy are present in the transcript. Any channel-level identity inference must rely on metadata not supplied here. ### Distinguishing Claims vs. Demonstrated Patterns | Creator claim | Observable demonstration | |---|---| | Speakers build efficiency infra at Meta (claim) | Talk demonstrates deep operational framing: 7-axis scheduling awareness, cascading failure loops, cost-per-successful-task economics (demonstrated pattern) | | Talk is a conference presentation (claim) | Transcript structure — co-speaker handoff mid-talk, live-demo hiccup ("Okay, this works I guess. Sorry") — confirms live event recording | | Industry needs an "inference control plane" (claim) | Talk consistently models inference as distributed-systems problem with queues, scheduling, fault isolation, and observability loops throughout | ### Notes for Synthesis - This single video suggests a channel surface (or at least content source) oriented toward **AI infrastructure engineering conference talks at production/hyperscale**, but no cross-video channel conclusions can be drawn from this material alone. - Transcript contains transcription artifacts ("infants talk" for "AI inference talk," "VLM" for "vLLM") — evidence should be cited cautiously where names of tools are involved. # Delivery Observation Report **Talk:** *Operating Distributed Inference Systems at Scale — Nishant Gupta & Naman Ahuja, Meta* --- ## Speaker Structure - **Two-speaker handoff**: The talk is split mid-way — Nishant handles framing, comparisons, scheduling, and optimization; the co-speaker takes over at metrics, observability, and closing lessons. Handoff is explicit: *"Now we'll hand it over to my co-speaker Naman to talk about the remaining talk."* - A small unscripted moment survives in the transcript from the second speaker: *"Okay, this works I guess. Sorry for..."* — suggesting a live-av or slide transition. ## Recurring Framing Habits - **"Takeaway" signposting**: Both speakers repeatedly mark key points: - *"The key to take away that you cannot plan capacity for the agents the same way we did for microservices."* - *"The takeaway is that the bottleneck is not just the model, it's the orchestration itself."* - *"So main takeaway is do not optimize for tokens. Optimize for successful task."* - **"Let's talk about" / "Now let's talk about" transitions**: Used frequently to move between sections — *"let's talk a little bit about agentic demand explosion"*, *"Now let's talk about scheduulers"*, *"Now let's talk about optimizations"*, *"Now let's talk about reliability"*. - **Rhetorical question-and-answer**: Ideas are often introduced as questions then resolved: - *"First, can we avoid the work? Meaning can we skip it entirely..."* - *"The operating question becomes how the platform know what to do next."* - *"Think about what happens if we have already streamed 200 tokens back to the user and suddenly a GPU host is preempted..."* ## Core Teaching Patterns - **Microservices-vs-inference contrast as the organizing spine**: Nearly every section runs a historical parallel (cloud era 2008 → AI era), covering request shape, batching, state, scaling units, and failure modes. Example: *"capacity planning was mostly a spreadsheet exercise"* vs. agentic capacity scaling. - **Enumerated dimensions and numbers**: Heavy use of concrete counts — *"seven axis"* of scheduling awareness, *"10 to 20 for co-pilots and 50 for research agents"*, *"50 tokens to 100,000 tokens"*, *"100 times more expensive... 10 times slower to acquire"*. - **Named frameworks**: The four-quadrant optimization frame (*avoid, share, move away, delay the work*) and the cost/latency/throughput triangle, with an analogy: *"It's pretty analogous to cap theorem."* - **Reflexive "we" positioning**: The speakers speak as operators of the system — *"what we're seeing in our workloads"*, *"a framework which at least I like to use"*, *"operating lessons we have experienced in a infra"*. ## Recurring Vocabulary - High-frequency terms: **orchestration / control plane, KV cache, admission control, prefill/decode, workload aware / agentic aware, SLO, entangled**. - Signature aphorisms (repeated in near-identical form by both speakers): - *"The intelligence might lie in the model but the economics the reliability and the user experience are all in the infrastructure."* - *"Optimize cost per successful task"* (stated by Nishant and again in the co-speaker's lessons). - *"Infrastructure is no longer a server problem. It's an orchestration problem."* - Emphatic doubling for stress: *"very very important"* (used twice), *"much much more"*, *"all that all those requests"*. ## Notable Stability vs. Uncertainty - **Stable-looking patterns**: takeaway signposting, microservice contrast structure, and the "cost per successful task" message recur across both speakers and multiple sections — high confidence these are deliberate habits. - **Thin evidence**: Emphatic repetition ("very very important") appears only twice; whether it is a habitual intensifier or situational emphasis cannot be confirmed from one talk. - The transcript contains apparent transcription artifacts (*"infants talk"*, *"bog, kubernetes, msos"*, *"shoulduling"*, *"disagreed inference"*); these are treated as transcription noise, not observed delivery features, and no conclusions are drawn from them. # Operating Distributed Inference Systems at Scale — Observation Record **Video:** *Operating Distributed Inference Systems at Scale* **Speakers:** Nishant Gupta and Naman Ahuja (transcript renders names as "Nishan Gupta" / "Naman Auja"), Meta — stated to work on efficiency for training and inference infrastructure. --- ## Part 1: Nishant Gupta — Inference as an Orchestration Problem ### Inference as hyperscale workload - **Claim:** Inference is no longer a research artifact; it is a foundational hyperscale infrastructure workload growing faster than any workload previously seen, and its traffic already outpaces the largest microservices in the world. ### Cloud-era analogy - **Claim:** The cloud era moved value up the stack: virtualization (VM offerings, ~2008) → schedulers (Borg, Kubernetes, Mesos — transcript: "bog, kubernetes, msos") → service meshes → autoscalers → platforms. **"The orchestration layer is what actually captured the value and the complexity."** - **Claim:** AI is on the same trajectory, compressed from a decade into a few years: simple GPU models → serving frameworks (vLLM, Triton — transcript: "VLM," "Triton") → an emerging orchestration layer addressing routing, KV cache management, prefill/decode disaggregation, and multimodel multiplexing. - **Framing:** In this next phase, the differentiator is not the best models or kernels but **"the whole ecosystem... the control plane and the orchestration."** ### Agentic demand explosion - **Claim:** Classical web serving scaled capacity roughly linearly with users; capacity planning was "mostly a spreadsheet exercise." - **Claim:** Agentic serving capacity scales with: **users × calls per user × tokens**, varying by model, optimizations, and hardware SKU. - Chatbot: ~1 model call per turn; co-pilots: 10–20; research agents: ~50; autonomous workflows: **thousands of calls with no human in the loop**. - **Takeaway (explicit):** "You cannot plan capacity for the agents the same way we did for microservices." Requires elasticity, workload-aware scheduling, and admission control. ### Microservices vs. LLM serving (stated dimension-by-dimension) - **Request shape:** microservices assume short, uniform requests; LLM requests in their workloads range from 50 to 100,000 tokens, with very different prefill vs. decode compute profiles. - **Batching:** classical stacking batching (if at all, at the load balancer) vs. required **continuous in-flight batching** — otherwise throughput collapses "by an order of magnitude." - **State:** classical microservices were stateless; LLM serving carries expensive per-request **KV cache** state — "very expensive to build and even more expensive to throw away." - **Scaling units:** cheap CPU pods vs. GPUs described as **100× more expensive and 10× slower to acquire**; casual overprovisioning causes huge waste. - **Failure mode:** crashed hosts/pods could be restarted and state rebuilt; LLM serving has slow cold-to-hot startup, and a GPU preempted mid-decode can drop thousands of in-flight tokens and cause queue buildup. - **Takeaway (explicit):** "The bottleneck is not just the model, it's the orchestration itself." ### Hidden decisions behind a prompt - Steps listed: authentication, model selection, region selection, admission control, cache lookup, GPU execution, batching. **Only prefill/decode requires the model**; everything else is infrastructure. - **Claim:** "The intelligence might lie in the model but the economics, the reliability and the user experience are all in the infrastructure." Platform teams now have more impact on product quality than before. ### Layer coupling - **Claim:** The layers are not new (20+ years old); **what is new is the combination and coupling between them.** - **Causal chain as stated:** a routing decision changes cache hit rate → changes batch composition → changes GPU utilization → changes autoscaling decisions. "Everything is entangled." Regressions must be diagnosed top-to-bottom. ### Inference as a distributed transaction - **Claim:** Each hop in the prompt path (gateway → router → cache lookup → scheduler → serving runtime → streamed response) can retry, time out, fall back, or fail, each with its own SLOs, while streaming to the user. **Partial-failure semantics are harder than regular RPC** — e.g., 200 tokens already streamed when a GPU host is preempted means you cannot simply retry. - **Claim:** "Reliability cannot be built at the edge. It has to be a property of the control plane because the control plane is the one which sees the whole workflow." ### Scheduling: seven axes of awareness - Traditional scheduling: bin packing across 3–4 dimensions (CPU, memory, etc.). - Inference scheduling must be aware of: **GPU type** (heterogeneous hardware, e.g., H100/B200-class with different network topologies), **HBM headroom**, **KV cache state**, **model weights** (warm vs. cold), **tenant priority** (multi-tenant SLO profiles), **workflow context**, and **latency budget**. - **Concrete example given:** a request at step 3 of 5 in a workflow that has already spent X + Y dollars — if step 3 fails, the workflow terminates and all compute is wasted. Hence workflow-aware orchestration changes admission decisions, priority, and retries. ### Four-quadrant optimization framework 1. **Avoid the work** — caching (prefix, response, semantic caching). 2. **Share the work** — batching (continuous batching, chunked prefill, speculative decoding). 3. **Move the work** — routing (smaller model, cheaper region). 4. **Delay the work** — admission control and queueing (priority classes, deadline-aware scheduling). - **Claim:** the framework transfers across stacks (vLLM, SGLang, TensorRT); every technique fits one quadrant. ### Cost economics - **Claim:** Cost is not just GPU/model cost — it includes retries, storage, failures, network, and operational cost. - **Position:** Do not optimize cost per token or per request; **optimize cost per successful task**, "because this is what actually users care about." ### Cascading failures - **Stated failure loop:** GPU degrades → latency rises → client retries → queue depth increases → healthy GPUs saturate → more retries → regional failures. - **Agentic twist:** KV cache cannot be casually restarted or rerouted; a cold pool must warm up while the hot pool absorbs everything. - **Prescribed loop breakers:** circuit breakers at the routing layer; admission control that **rejects rather than queues**; load shedding tied to **queue depth** (not just CPU/memory); and deliberate retry budgets — otherwise cost scales quickly. --- ## Part 2: Naman Ahuja — Control Plane, Metrics, and Lessons ### Inference as a distributed system - **Claim:** At production scale, inference resembles a classic distributed system (queues, scheduling, autoscaling, fault isolation) but with new constraints: **CPU, HBM, KV cache, and cost per successful task** instead of CPU/memory alone. - **Framing of observability:** "not just dashboards" — telemetry is an **input signal to the control loop** driving scheduling/routing decisions, in a continuous sense–detect–adapt cycle. ### Four key metrics (as listed) - **Time to first token** — responsiveness of first response. - **Utilization ratio** — whether memory or compute is the bottleneck. - **Success per dollar** — whether the platform is delivering efficiently. - **End-to-end request latency** — time across the full request path. ### Latency–cost–throughput trade-off - **Claim:** A core trade-off exists "analogous to CAP theorem" — you cannot get all three: - Larger batches → better throughput/cost, worse tail latency. - Speculative decoding → better latency, extra compute → higher cost per token. - Smaller models → lower latency and cost, lower quality → failure analysis and retries bring cost back up. - **Position:** "Every serving decision moves the system somewhere in this triangle. And our job is to find the perfect setting. It's just an optimization problem." ### Inference control plane - **Claim:** Routing, batching, caching, scheduling, and reliability are converging into a logical layer — the **inference control plane** — analogous to how Kubernetes turned VM-era autoscaling/scheduling into a control plane. - **Claim:** Models are becoming resources; GPU, KV cache, tokens, latency, and cost are what get scheduled around; the control plane decides which model serves which request and how it is batched. - **Practical stance:** whether built internally, open source, or vendor-provided, **"the key design is to assume the layer will exist."** ### Operating lessons (as enumerated) - Infrastructure bottlenecks usually show up **before** model bottlenecks in production (often scheduling/routing/capacity breakdowns, not inference itself). - **Elasticity is needed** — more GPUs just hide the problem. - **Scheduling decisions overpower efficiency** — the same fleet performs very differently depending on scheduling and batching. - **Control loops beat manual processes** — the platform must sense, detect, and adapt automatically. ### Closing theses - **"Do not optimize for tokens. Optimize for successful task."** - **Three phases of AI infrastructure (as stated):** (1) better models/benchmarks, (2) current: faster inference, lower latency, better batching/GPU utilization, (3) next: **orchestration**, where GPU, memory, and cache are just resources to be scheduled and controlled. Teams that understand this early will build infrastructure for the future. - **Closing line:** "Infrastructure is no longer a server problem. It's an orchestration problem." --- ## Notes on Source Fidelity - Speaker attribution for Part 2 rests on the transcript's explicit handoff ("we'll hand it over to my co-speaker Naman"). - The transcript contains frequent transcription errors (e.g., "VLM" for vLLM, "HPM/HVM" for HBM, "QEP" for queue depth); interpreted terms are flagged in brackets only where meaning is clear from context. - No contradictions between the two speakers' positions were present; their claims are complementary, though their framings (framework quadrants vs. trade-off triangle) remain distinct as expressed. # Video Attribution & Provenance Record ## Speaker Identification & Roles | Speaker | Role | Stated Affiliation | |---|---|---| | **Nishant Gupta** | Co-speaker (primary presenter of first portion) | Meta — efficiency training and inference infrastructure | | **Naman Ahuja** | Co-speaker (delivers second portion of talk) | Meta — same team | - Both identities are stated self-introductions in the transcript: *"My name is Nishan Gupta and I'm joined today by my co-speaker Naman Auja"* (transcript contains spelling variants "Nishan Gupta" and "Naman Auja"; title metadata gives "Nishant Gupta & Naman Ahuja"). - Setting: talk at **AI Engineer World's Fair** ("the first infants talk on the last day of AI engineer world fair" — transcript contains transcription errors). - No narrator, guests, or other speakers identified. Handoff occurs mid-talk: *"Now we'll hand it over to my co-speaker Naman."* ## Referenced Sources & Third-Party Claims Claims below are asserted by the speakers; no external citations or sources are provided in the transcript. - **Internal experience claims (Meta):** inference traffic "already outpaces the largest microservices in the world"; workload-specific figures (50–100,000 token request variance; 1 model call/turn for chatbots, "10 to 20 for co-pilots and 50 for research agents"). - **Named third-party tools/technologies (as industry context):** vLLM ("VLM"), SGLang ("SD lang" / "SDG lang"), TensorRT, Triton, Kubernetes, Borg, Mesos ("msos"), NVIDIA H100/B200 and AMD hardware references. - **Historical analogy:** cloud era circa 2008 (VMs → schedulers → service meshes → autoscalers) offered as a speaker framing, not a cited source. - **Framework claims:** a four-quadrant optimization framework (avoid / share / move / delay work) and a latency-cost-throughput tradeoff analogy to CAP theorem — presented as the speakers' own framing. ## Promotional Segments - **No promotional segments flagged.** No product pitches, vendor offers, or calls to action appear in the transcript. The talk references Meta's work descriptively ("infrastructure at Meta") without promoting a specific product. ## Comment Availability - **Not determinable.** The transcript and metadata provide no information about comments, Q&A sessions, or audience interaction availability. ## Provenance Limitations & Uncertainty - Transcript contains systematic transcription errors (e.g., "infence" for inference, "shoulduling" for scheduling, "KV cache" spelled "KVB/QEP" variants, "H00/E00/B200" for hardware names). Speaker names standardized via title metadata. - No timestamps, slides, or external references are supplied; all claims are attributed solely to the two speakers' spoken statements. - Audience Q&A portion is not present in the transcript. Good morning everyone. Welcome to the first infants talk on the last day of AI engineer world fair. My name is Nishan Gupta and I'm joined today by my co-speaker Naman Auja. We work on building the efficiency training and inference infrastructure at Meta. Today we're going to be talking about how do you operate distributed infence systems at scale. As we all know, inference is no longer just a research artifact boiled to a product. It's a foundational hypers scale infrastructure workload which is growing at a tremendous rate. The infence traffic already outpaces the largest microservices in the world and the rate of growth is fastest of any workload we have ever seen. Let's rewind back to around 2008 and try to compare the AI era with the cloud era. In around 2008, the cloud started as virtual machine offerings. The interesting injuring was virtualization. Then over time, the value moved up to the stack to the scheduulers. Think bog, kubernetes, msos, then to service meshes, then to autoscalers, and then to various platforms that were built on top of it. The orchestation layer is what actually captured the value and the complexity. AI is on the exact same trajectory but just compressed into a last few years instead of a decade. We started with simple models running on GPUs. Then we saw the evolution of model serving frameworks like VLM to serve Triton and now we are watching the orchestration layer emerging in real time addressing the complex challenges of routing KV cache management prefield decode disagregation and multimodel multiplexing. In this next phase of AI era, it's not just about the best models or the kernels or the optimizations. It's about the whole ecosystem. It's about the control plane and the orchestration. And this is what we're going to be focusing in this today's talk. So let's talk a little bit about agentic demand explosion. In the classical web serving before the AI inference workloads kicked off the capacity scaled roughly linearly with users depending upon the workload type double the users mostly double the QPS double the infrastructure fleet if there are no optimizations and capacity planning was mostly a spreadsheet exercise. In this new agentic serving capacity is scaling with number of users time number of calls per users times number of tokens which varies depending on the model the optimizations the hardware SKU you have a chatbot can have one model call per turn which is evolving now with 10 to 20 for co-pilots and 50 for research agents and now it's thousands of such calls for autonomous workflows with no human in the loop. The key to take away that you cannot plan capacity for the agents the same way we did for microservices. We need to think about elasticity and implement workload aware scheduling and admission control. Now let's try to dive a little bit deep into the differences pros and cons differences between the traditional microser in serving versus the modern infence serving across these key dimensions. The request shape microservices assumes short uniform request wherever the LLM requests we are seeing in our workloads they can vary from 50 tokens to 100,000 tokens with vastly different compute profiles between prefill and decode stage for batching classical stacks for microservices was doing stacking batching mostly at the load balance layer if at all however LLM serving requires continuous inflight batching how otherwise the throughput collapses by an order of magnitude or state most of the classical microservices were stateless when we not talking about storage layer however LLM serving does require a huge per request state the KV cache which require which is very expensive to build and even more expensive to throw away scaling units when we think about traditional microservices we could run them in cheap CPUs in pods however for model inferencing we require to run on GPUs which are 100 times more expensive which are 10 times slower to acquire and we cannot overprovision them casually otherwise it will lead to a huge wastage. failure mode. When you think about traditional microservices like most of us have built over the last couple of years, we could even if a micro even if a host crashed or a bot crash, we could restart it. We could rebuild the state if needed be. However, for the model inferencing, it takes a huge amount of time to go from cold to hot startup and these if and if a GPU is mid decode, it can drop thousands of in-flight tokens and it can lead to a queue buildup. The takeaway is that the bottleneck is not just the model, it's the orchestration itself. Now, as we can see in these hidden decisions behind a prompt, when we go to any gentic application, it requires a bunch of steps which are behind the scenes. We have to authenticate. We have to choose a model depending on the request type. We have to select a region where it goes, we have to do admission control. We have to do caching lookup. We have to do the we have to run it on the GPU. We have to do batching. And there's a bunch of other steps involved. And as we can see out of all these steps only one step requires the model which is the pre-filled decode. If it's disagreed inference or just if it's not disagreed inference other steps require infrastructure. The intelligence might lie in the model but the economics the reliability and the user experience are all in the infrastructure. And this is why a lot of platform teams across a lot of companies are having much more impact on the product quality and the success ra much more than before. Now most of us in this room have a deep expertise across one or two or three layers. We might own kernels or kernel optimizations. We might own routing or the product itself or we might be operating the GPU infrastructure or the cluster itself. But very few of us have operated the whole stack or seen thought about it end to end. As you can see in these layers, these layers are not new. They have been around for 20 years or more. What is new is the combination and the coupling between them. A decision at the routing layer can change the cache hit rate at the model layer which can change the batch composition which can change the GP utilization which can change the autoscaling decision because of the change in GPUs simplization. So everything is entangled when we have any regressions in our inference workloads. It's not just about understanding what happened at the caching layer admission control. We need to think about the stack top to bottom. And when we see any bottleneck, it's very very important to understand at which layer is that bottleneck so that we can invest properly. Now diving a little bit deep into a how a prompt works when we have a prompt for any application be it just if you want to generate an image or if you have a research task or if you have complex multi-agent orchestration more or less it involves a bunch of these steps. The prompt goes to the gateway. Then it goes to the router after which it does the cache lookup if the request was already seen before. Then it goes to the scheduulers which decides on which GPU cluster it should run on which hardware it can be on Nvidia or AMD or your in-house silicon chip which then goes to the appropriate serving runtime VLM SD lang or whatever we have we are working on and then we stream the response back to the user according to the SLO profiles of time to first token and time between each token and while making sure the throughput is what the users desired. Now as we can see this inference behaves like a distributed transaction. Each arrow in this diagram is a network hop. Every of every one of these hops can retry. It can time out. It can fall back. It can even fail. And each of these hop will have an SLOs's and it will it it is streaming back to the user. So if it fails, the partial fail is mantics are much more harder to deal with than if for a regular RPC call. Think about what happens if we have already streamed 200 tokens back to the user and suddenly a GPU host is preempted due to a scheduled or a planned or unplanned maintenance event. We cannot just retry. We have to think about it holistically. This is where this is why reliability we cannot build reliability at the edge. It has to be a property of the control plane because the control plane is the one which sees the whole workflow. Now let's talk about scheduulers and some of the optimizations uh and how we can think about it. So for the traditional microservices we used to think about bin packing of traditional microservices across three four dimensions could be across resource utilization CPU memory or could be across all domains depending upon if you're using AWS if you're using your own in-house cloud providers but for inference the schedule needs to be aware AC of across at least seven access when we schedule a particular request it has to be aware of the GPU type there can be any number of heterogeneous hardwarees in your cluster H00 versus E00 versus B200 with different network topologies. It has to be aware of the HPM headroom, KV cache state, the model weights, whether they are already loaded, whether they're cold or we have to whether they have already warmed up, we have to cold start it. It has to be aware of the tenant priority. There can be any number of tenants running on that multi-tenant cluster with different SLO profiles. We have to also be aware of the workflow context. Are we in the third step of reasoning that has already spent X dollar or are we at the initial stages and we can terminate the workflow if it if we are overprovision. We have to also think about latency budget depending upon the type of agentic application we are building. So this brings us to the idea that we have to make sure that we implement agentic aware scheduling. We have to make sure that we place the work which will which will finish the in the fastest and the cheapest time as opposed to just placing it on a random GPU. A concrete example might be that the skiller needs to be aware that request R is at step three of a five in a workflow and it has already and step one and two has already spent X plus Y dollar. So if a step three fails the whole workflow will be terminated and we have wasted all that compute resources. That's why workflow aware orchestration is very very important because it it will change the admission decisions the priority and how we retry. Now let's talk about optimizations. I will not go too deep into a lot of optimizations. There's a lot of research which has already been done outside. But I would like to share a framework which at least I like to use when it comes to it. And uh we can place it into four quadrants. First, can we avoid the work? Meaning can we skip it entirely through caching through techniques like prefix caching, response caching, semantic caching. The second can we share the work? Can multiple requests share compute through batching? Think continuous patching, prefill decode, chunk prefill, specular decoding. Third, can we move away the work? Can we send it somewhere to a cheaper model or closer to the user through mostly routing? Can we route it to a smaller model or a cheaper region or some other techniques? And lastly, can we delay the work? Can we wait for a better moment through admission control and queueing which requires us to understand the priority classes of these requests and implement deadline aware scheduling. Now this framework is very powerful because it transfers to various stacks. You might be using VLM or SDG lang or tensor RT but every technique fits into one of these quadrants. So whenever we think about any optimization to our model we have to apply we have to do a comparison and contrast with the previous techniques and see how all these stack with each other. Now whenever we think about scale it's not just important to think about the performance of the model. We have to think about the cost economics as well. This is where it's important to understand what metric we're trying to optimize because the cost is not just the cost of the GPU or the model. It is the cost of all these parameters retries, storage, failures, network and of course the operational cost of development and all that stuff. What is important is to understand what is the key key performance indicator for your product which will add value to the users. So it's not important to optimize just cost per token or cost per request. We have to optimize cost per successful task because this is what actually users care about. And if you're able to optimize that, the cost for the overall product decreases and the users are much more happier. Now let's talk about reliability a little bit about reliability and uh what it means to prevent cascading failures. Now the failure story is never a GPU is preempted or GPU has died. The interesting stories is the feedback loop that follows. So a GPU can degrade, the latency can rise, the client retries, the Q depth increases, the healthy GPUs will saturate which will follow more retries, much more more full regional failures. This is the classic cascading failures. But for with a twist for agentic application, the KV cache, we cannot just casually restart or reroute to a different cluster. A cold pool has to warm up before it can absorb traffic during which the hot pool has to take on all that all those requests. This is why it's important to design the loop breakers very deliberately the circuit breakers at the routing layer the admission control which rejects rather than just queuing the load shedding tied to QEP not just CPU or memory utilization and we have to also think about retry budgets because if you don't think about all these things the cost can scale much much more quickly. Now we'll hand it over to my co-speaker Naman to talk about the remaining talk. Okay, this works I guess. Sorry for so once inference reach a production scale it starts looking much more like a distributed system. We are not longer just calling a model. It's more like a classic distributed uh system problem. So in distributed system we talk about cues, scheduuling, autoscaling, fault isolation. These are some of the dimensions. Inference has all these problems. But there are new constraints. Now instead of CPU, memory alone, we have CPU, HVM, KV cache and course per successful task. So the operating question becomes how the platform know what to do next. And that's where observability comes into play. It's not just about dashboards. It's about uh how to provide input signal to the control loop telemetry fields analysis analysis drive decisions is change of shoulduling and routing and finally we just rinse and repeat um I'll just give an overview what are some important metrics first one is time to first token which tells me about uh how much time does it really take to get the first response then we have utilization ratio which tell me whether memory or compute is the bottleneck we have success per dollar that tests us whether platform is actually delivering and working efficiently. And finally, we have end to end race latency that tells us what's the time being spent across the full request path. There's a core trader between latency, cost, and throughput. You cannot just get all of them. It's pretty analogous to cap theorem. If I increase the batch size, um I improve the throughput and cost efficiency, but I may hurt tail latency. If I use speculative decoding, I may improve latency, but there are some extra compute. Ultimately, you know, we increase the cost per token. And finally, I can use a simple model, smaller model. I can reduce the latency and cost, but the response will be of low quality. Ultimately, I'll do failure analysis, do retries, which brings the cost back up. So, every serving decision moves system somewhere in this triangle. And our job is to find the perfect setting. It's just an optimization problem. Now this is where the industry is heading right now. Inference needs its own control plane. Everything we discussed routing, batching, caching, scheduuling, reliability, they cannot be a separate knob. Now they're converging into a logical layer. Let's call it inference control plane. We used to manage VMs before in distributed system. We have autoscaling scheduulers and we had Kubernetes which turned these into control plane. Inference is going through the same transition now. uh models are becoming resources. GPU, KV cache, token, latency, cost are now scheduled around. The control plane decides which models serve which request and how it is batched. So whether we build this layer internally, use open source or from a vendor, the key design is to assume layer will exist. Now let's discuss about where are some of the operating lessons we have experienced in a infra and how they are applicable here. The first lesson is that infrastructure bottlenecks usually show up before model bottlenecks in production. Many failures can come but there can be just aboutuling and routing or capacity breakdown. So these are not related to inference. It's about infrastructure problems. Then we have elasticity. We need elasticity in a system. We can have more GPUs but this will not really solve the problem. We are just hiding the problem. Then our solution to schedule the decision overpowered the efficiency. The same fleet can deliver very optimally depending on how you're shoulduling it or how you're batching it. Then we have control loops which beats the manual process. The platform has to sense, detect and automatically adapt to the system. So main takeaway is do not optimize for tokens. Optimize for successful task. So this is a broader shift I want to leave you with. The first phase of a infrastructure was about better model. We invest a lot of time in improving our models, smarting, making the model smarter and coming with better benchmarks. The current phase right now is faster inference, lower latency, better batching and better GP utilization. But the next phase is about orchestration. That means GPU, memory, cache and everything. These are just resources and they need to be scheduled and controlled. The teams that understand this early on will build infrastructure for the future. So the closing idea is infrastructure is no longer a server problem. It's an orchestration problem. Thank you.