A conference talk by Qianru Lao and Lu Zhang (OpenAI) on routing LLM inference traffic in production. It covers the limitations of an earlier approach where engine-reported signals fed a proportional controller that adjusted routing weights—causing unexplainable weight assignments, coupled tuning behavior, and oscillation that disrupted KV cache locality—and the replacement architecture: a control plane with a global fleet view paired with per-cluster data planes that route requests from cached weight snapshots. Signals feed an optimizer that minimizes expected end-to-end latency (accounting for network distance and engine-side queueing) under constraints that all requests are routed and no engine exceeds capacity, illustrated by an example where a farther, less loaded engine beats a nearer overloaded one. Protective mechanisms such as outlier penalties, utilization-aware retry budgets to prevent retry storms, and load shedding are also discussed. # Video Identity Observations ## Source Context - **Video**: *"Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI"* - **Speakers**: Two engineers who state, "we both work on the inference team at OpenAI." ## Intended Audience The talk addresses a **technical, production-engineering audience** rather than a general or beginner audience: - Assumed familiarity with LLM inference internals is stated directly: *"as AI engineers we all kind of know that production in many cases are not behaving in the most ideal case."* - Concepts are introduced with "as you may already heard" framing (e.g., TTFB/TTFT, token throughput, KV cache): *"there's a important concept of a KV cache which is also well known"* — implying the audience already knows these terms. - Control-theory and distributed-systems vocabulary (P controllers, consistent hashing, control plane/data plane, retry storms, load shedding) is used without extended tutorial framing, with only brief asides like the joke *"this P controller will not help you kill a Linux process."* ## Claimed Scope This is a **conference-style case study talk**, explicitly scoped rather than a general channel topic: - The talk covers OpenAI's inference load balancer evolution: *"how our system evolved from routing based on feedback loops driven by engine signals to a more explicit and a predictable policy."* - A stated agenda: load balancer intro → control/data plane architecture → a case study on reducing *"global network overhead"* → protection mechanisms (penalties, retry budgets, load shedding). - Discusses internal production architecture at a design level (CPU front-end clusters, GPU engine clusters, optimizer inputs/outputs, three system paths), stopping short of code-level detail. ## Observable Presentation Patterns - **Insider practitioner framing**: The creators present themselves as operators of the system, not commentators — *"we don't need to do a lot of manual intervention and it just works"* (describing the early system), and frank admission of its trade-offs (oscillations, hard-to-reason-about decisions). - **Educational structure**: Definitions, worked examples, and a numeric capacity example (regions serving 90/100, 120/100, 40/80 RPS) suggest pedagogical intent despite the expert audience. - **Event context**: The closing — *"both of us will be around in our booth area this afternoon"* — indicates this was recorded at a live conference with a booth presence, i.e., an invited talk rather than standalone channel content. ## Claim vs. Demonstration Distinction - **Creator claims**: The new architecture is "more explicit and predictable," globally optimized, and reduced network overhead (per title and agenda). - **What is demonstrated**: The conceptual architecture (asynchronous weight publishing, synchronous local selection), the optimizer's four inputs and hard constraints, and the three protection mechanisms — all at design-description level, without performance numbers or benchmarks being stated in the transcript. ## Notes for Channel-Level Synthesis This single video evidences a channel (or uploads) of **expert-level infrastructure/systems talks by industry practitioners**, likely conference recordings, focused on LLM serving at scale. No broader channel self-description (e.g., channel tagline, series branding) is present in the supplied material; audience and scope inferences here are limited to this video's content. # Delivery Observation: Routing LLM Inference in Production (Qianru Lao & Lu Zhang, OpenAI) ## Speaker Handoff Structure - The talk is explicitly **co-presented with named handoffs**: > "I'm going to hand over to Chenu to uh deep dive into the new architecture we tried out." > "Thanks Chenu. So Lou will continue to talk about the protection mechanisms in the system." - Each speaker opens their section with a transition marker ("Yeah, thank you L. So I'm going to talk about...", "So that's pretty much of my part"). ## Recurring Teaching Habits - **Question-driven segmentation.** Both speakers repeatedly pivot sections around a rhetorical question: > "Then the important question becomes where are the WS come from." > "But that still leaves one question. How do we compute those routing weights? But before answer that question, let's answer another question first. Why not just send a request to the nearest engine?" > "So how do we keep our production system uh heresy as much as possible under the heavy load?" - **"What is X / where does it sit" openings.** The talk begins with definitional grounding before complexity: > "So to begin with, what is the inference load balancer and where it sit?" - **Escalating-baseline teaching.** A naive solution is proposed, then dismantled step by step: > "One most naive baseline might be round robin... But if you think a little bit more that doesn't make sense... A probably better solution might be... But that's not enough either." - **Concrete micro-examples with numbers.** The nearest-only routing argument is taught via a three-region toy scenario with explicit figures ("CPU cluster A send 90 RPS and the nearby engine A can serve 100 RPS"). - **Explicit signposting of summaries.** > "In summary, there are three important paths through the system." > "That's pretty much concludes our talk today." ## Vocabulary and Terminology Patterns - Heavy, consistent use of **signal/latency acronyms** introduced with glosses: > "time to first token TTF time between output tokens also known as token throughput or time between tokens" - **Anthropomorphized system behavior** for intuition: > "the engine turns a bit cooler and this signal get fit to the controller. The controller now thinks that hey this engine can take a lot more traffic." - Recurring framings of **trade-offs**: "comes with big trade-offs," "it has a lot of nice properties... However," "That's why we need something better than the nearest only routing." - Casual audience-alignment phrases: "as you may already heard," "as AI engineers we all kind of know that production in many cases are not behaving in the most ideal case." - **Humor aside** when introducing the P-controller: > "no this P controller will not help you care a Linux process" (apparently "kill a Linux process" — a pun on P-controller vs. `pckill`/`pkill`). ## Structural Communication Patterns - **Diagram-first exposition**: components located spatially before function is explained ("On the left hand side are the front end clusters... on the right hand side are the engine clusters"). - **Numbered/enumerated responsibilities**: "It has two main responsibilities select an engine and proxing the request"; "The optimizer accepts four types of input"; "it need to respect several hard constraints. First... Second... Third..." - **"Why" before "how"**: each mechanism is motivated by a failure mode before the design is revealed (feedback-loop oscillation motivating the new architecture; retry storms motivating retry budgets). - Synchronous-vs-asynchronous distinction repeated for emphasis: > "So only the first pass is synchronous but it's fast... The other two loops are asynchronous loop." ## Notable Isolated Moments (lower confidence as stable patterns) - **Self-deprecating framing of recency**: > "And to be honest, early days in in this industry sounds a lot more historic than it really is." - **Industry-context humor**: > "as you may already heard nowadays GPUs are pretty popular and expensive." - These occur once each; whether they reflect habitual delivery style cannot be established from a single talk. ## Delivery Characteristics - **Spoken-language disfluencies** are frequent and unpolished in transcript form: repeated words ("less constraint less constraint request," "a problems," "way" for "weight"), consistent across both speakers — suggests live, unrehearsed-reading delivery. - **Closing is invitational and logistical**, giving the audience a concrete next step: > "both of us will be around in our uh booth area this afternoon. So if you have further questions, feel free to walk uh to the area and and chat with us." ## Uncertainty Notes - Only one talk is available; recurrence is judged within a single ~18-minute delivery, so patterns like humor or anthropomorphizing could be topic-driven rather than habitual. - Some quotations contain apparent transcription artifacts ("IRB," "heresy," "basier"); these are reported as they appear in the supplied transcript, not corrected. # Routing LLM Inference in Production: From Engine Signals to Policy **Speakers:** Lou (per transcript; listed in title as Lu Zhang) and Chenu (listed in title as Qianru Lao), OpenAI inference team. --- ## 1. System Overview — Lou ### The Inference Load Balancer (IRB) - **Architecture layout:** - Left side: **front-end CPU clusters** acting as gateways; receive user requests and prepare them as inference requests. - Right side: **engine clusters**, usually GPU clusters, each hosting multiple inference engines. - Middle: the **IRB** — runs on the front-end clusters and acts as "a bridge into our inference stack." - **IRB's two responsibilities:** *select an engine* and *proxy the request*; this talk focuses on engine selection. ### Similarities to and Differences from Traditional Load Balancing - **Like traditional load balancers:** a request targets a model backed by multiple engines, which may live across clusters, regions, or continents — providing resiliency against localized degradation or cluster failures. - **Inference-specific nuances:** - Real-time signals: **TTFT** (time to first token), **TBT** (time between tokens / token throughput), health and utilization. - **KV cache locality:** follow-up turns of a conversation benefit from going to the engine already caching that context — avoids recomputation, improves efficiency, reduces latency. - **Framing claim (Lou):** "the combination of performance reliability locality cache awareness is what makes it such an interesting problem." --- ## 2. The Early Architecture: Feedback-Loop Weights — Lou ### Mechanism 1. **Constraint filtering:** not every engine can serve every request (capability constraints, compute or data-residency restrictions). 2. Among remaining engines, IRB used **weighted consistent hashing** to pick the destination. 3. **Weight generation via periodic feedback loop:** engines report signals; a controller periodically smooths them and computes a **performance score**, compared against the **fleet average**. Weights rise when performance beats the fleet average, fall when worse. 4. Lou characterizes this as conceptually similar to a **P controller** from classical control theory — explicitly noting the joke that it is not the kind of P controller that manages Linux processes — with only the proportional part borrowed. ### Claimed Benefits - Combines many signals into a single routing decision. - Adapts to observed performance; capacity-constraint differences self-balance (engines serving more request types receive relatively less of the constrained kinds, fitting back into the loop). - Reduced need for manual intervention: "it just works" (Lou). ### Stated Trade-offs and Limitations > "that kind of adaptability comes with big trade-offs" — Lou - **Opacity:** hard to reason about why a particular engine got a higher weight than expected; fine-tuning one aspect "is almost impossible to not impact something else." - **Uneven load:** models served by engines on different GPU scales have different characteristics, making distribution trickier. - **Bad oscillations from the feedback loop:** shifting traffic away cools an engine; the controller then reads it as able to take more traffic; traffic shifts back and forth between a few engines, **disrupting KV cache utilization**. - These limitations "motivated us to rethink about the architecture" (Lou). --- ## 3. New Architecture: Control Plane + Data Plane — Chenu ### Core Question and Baselines Rejected - The load balancer answers: *for each request from a CPU cluster, which engine should serve it?* - **Round robin rejected:** engines are not homogeneous (different hardware, capacity, health, distance); it also breaks cache locality. - **Per-cluster local best-choice rejected:** multiple CPU clusters routing independently to the same engines could overload them while others sit underutilized. - **Conclusion (Chenu):** a **globally optimized** solution is needed — a **control plane** with a global view computing optimized routing answers, and a **data plane** making fast per-request decisions using those answers. ### Responsibilities - **Data plane:** - Engine selector reads **local routing state** (candidate engines + weights), refreshed asynchronously — no per-request control plane consultation. - Collects real-time engine signals (number of ready replicas, engine health, etc.) for fast local guardrails and to surface to the control plane. - **Control plane:** - Combines live signals and network overhead with **offline regressions** of capacity, TTFT, and TBT; the **optimizer** turns these into routing weights, which it publishes for data planes to pull. ### Three Paths Through the System (Chenu) 1. **Inference request path** — synchronous, fast, purely local within the CPU cluster's data plane. 2. **Engine signal path** — asynchronous; both planes consume real-time signals (control plane for global optimization, data plane for local guardrails). 3. **Routing weight path** — asynchronous; control plane computes/publishes, data plane pulls into local cache. Chenu's summary claim: only the request path is synchronous; the other two loops "are to improve future routing decision." --- ## 4. The Routing Optimizer — Chenu ### Why Not Nearest-Only Routing? - Traffic demand and GPU capacity are **not geographically balanced**. Chenu's worked example: - Region 1: cluster A sends 90 RPS, nearby engine A serves 100 RPS → nearest-only is fine. - Region 2: cluster B sends 120 RPS, engine B serves 100 RPS → 20 RPS would wait on an overloaded engine. - Region 3: engine C at 40 of 80 RPS has 40 RPS spare. - Sending the extra 20 RPS to the farther engine C adds network distance but can avoid larger engine-side queuing — "a further engine might be faster end to end." ### Optimizer Inputs, Output, Goal, and Constraints - **Four inputs:** (1) requests per CPU cluster, (2) network latency to each engine, (3) available engine capacity and health, (4) TTFT/TBT latency profiles describing how engine-side latency changes with load. - **Output:** routing weights — the fraction of each CPU cluster's traffic that should go to each GPU engine. - **Goal:** minimize expected **end-to-end latency** across all routed traffic, where end-to-end includes both network distance and engine-side latency. Consequence: nearby engines attract traffic while they have headroom; farther engines win when nearby ones are near full. - **Hard constraints:** all traffic demand must be routed; engines stay within effective capacity; routing weights remain non-negative. --- ## 5. Protection Mechanisms — Lou ### Context Claim - Production is often non-ideal: clusters fail, GPUs/nodes degrade, networking gets "all kind of mysterious issues." ### Three Mechanisms 1. **Penalties:** detect outlier engines and reduce their routing weight, giving them a chance to recover from transient issues, or enabling human intervention / faulty-hardware replacement. 2. **Retries with dynamic caps/budgets:** retries are common but can worsen problems near system tipping points — more load → more failures → more retries (the "infamous retry storm"). Caps keep retries in an acceptable region, and the caps **must be dynamic**: more retries are tolerable in normal times than under heavy utilization. 3. **Load shedding (last resort):** when capacity cannot meet demand, proactively shed a portion of traffic so the system "degraded gracefully" rather than failing entirely. --- ## Speaker Attribution Note - Speaker names are taken from the transcript's self-introductions ("I'm Lou and this is my colleague Chenu") and the video title ("Qianru Lao & Lu Zhang, OpenAI"); the transcript renders names phonetically ("Lou," "Chenu"), which differ from title spellings. Attribution below follows the transcript's own speaker labels. # Video Attribution & Provenance Record ## Video Identity - **Title:** Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI - **Video ID:** `01a15ca7-5b2c-7dc4-9c50-e7b54c65aaf0` ## Speakers & Roles | Speaker (as identified in transcript/title) | Role | Notes | |---|---|---| | **Lou (Lu Zhang)** | Co-presenter, OpenAI inference team member | Introduced both speakers; presented agenda, load balancer overview, early feedback-loop routing, and protection mechanisms; name is rendered inconsistently in the transcript ("Lou," "L"); title metadata says "Lu Zhang" | | **Chenu (Qianru Lao)** | Co-presenter, OpenAI inference team member | Delivered the deep-dive on new control plane/data plane architecture, routing weight optimization, and network-overhead case study; name rendered "Chenu" in transcript; title metadata says "Qianru Lao" | > ⚠️ The transcript's phonetic spellings ("Lou"/"L," "Chenu") differ from the title's "Lu Zhang" and "Qianru Lao." Which name maps to which speaker cannot be confirmed with certainty from the transcript alone; identity attribution relies on the supplied title metadata. - **Narrator:** None — both speakers present as employees of OpenAI (self-identified: "we both work on the inference team at OpenAI"). - **No guests or third-party interviewees** appear in the transcript. ## Referenced Sources & Claims All claims originate from the two presenters describing their own system at OpenAI. **No external citations, papers, benchmarks, or third-party sources are referenced.** Key claims attributed to the presenters: - **Lou:** Description of the inference load balancer (ILB), early weighted-consistent-hashing routing, periodic feedback loop described as analogous to a P (proportional) controller, and its trade-offs (opacity, oscillations, KV cache disruption). - **Chenu:** Control plane/data plane architecture, three system paths (inference request, engine signal, routing weight), a worked nearest-vs-further routing example with illustrative RPS figures (Region 1/2/3), and the optimizer's inputs, constraints, and goal (minimize expected end-to-end latency). - **Lou:** Protection mechanisms — penalties, retry budgets/caps (retry storm mitigation), and load shedding. > Note: Illustrative numbers (e.g., "90 RPS," "100 RPS") are presented as hypothetical examples, not measured production figures. ## Promotional Content - **No product promotions** identified. - One **availability note** (not promotional): both speakers state they will be available at a "booth area this afternoon" for follow-up questions. ## Comment Availability - **Not specified.** The transcript contains no information about comments being enabled, disabled, or moderated. ## Evidence Channel Limits - Only the transcript was supplied; no curated metadata beyond the title was available. - Speaker-name spellings in the transcript are inconsistent/phonetic, so name-to-speaker mapping is uncertain. - No timestamps, slides, or visual content were provided. Hi everyone, thanks for joining our talk. I'm Lou and this is my colleague Chenu. So we today we're going to talk about the in uh we both work on the inference team at OpenAI and today we are going to talk about routing IM inference in production specifically how our system evolved from routing based on feedback loops driven by engine signals to a more explicit and a predictable policy which is still informed by engine signals. However, it's more like the way we use it is different. So uh for the agenda today we're going to begin by introducing the inference load balancer what it is what it does and how it has evolved and then Chenu will walk us through the newer control plane and data plane driven architecture uh what are the responsibilities of each and followed by a concrete case study of how we reduce the global network overhead and in the end I return to discuss the protection mechanisms that help keep the system stable and production level stress. So to begin with what is the inference load balancer and where it sit? So this is a very high level diagram of the system we are talking about. On the left hand side are the front end clusters. Those are the GPU cluster. Sorry, those are the CPU clusters that act as gateways into our system and they receive user requests then prepare them into the inference request that can be processed by the inference engines and on the right hand side are the engine clusters uh which are usually GPU clusters and each hosting multiple inference engines. So that's why we they got the name of engine clusters and as you may already heard nowadays GPUs are pretty popular and expensive. So um sitting in the middle it is the IRB or inference load balancer. It actually runs on the front end clusters but is also a bridge into our inference stack. It has two main responsibilities select an engine and proxing the request. For this talk, we are going to focus on the engine selection part. So in some ways, IRB resembles a very traditional load balancer because a request usually targets a model and a model is backed by multiple engines. They may live on different clusters in different regions or even across the continents because that gives us a good resiliency towards localized degradation or cluster failures. However, the inference uh stack or the uniqueness of the inference introduces a lot of nuances like uh it has to consider a bunch of signals reported in real time like the well-known time to first token TTF time between output tokens also known as token throughput or time between tokens and other hairness and utilization signals. Besides there's a important concept of a KV cache which is also well known but for example when the conversation already has a lot of the useful context cached in one engine sending the follow-up turns of the same conversation back to the same engine will avoid recomputation improve efficiency and uh reduce latency. So the combination of performance reliability locality cach awareness is what makes it such an interesting problem. Uh so how we attempted in the problem? Let's take a look at the early days. And to be honest, early days in in this industry sounds a lot more historic than it really is. And the routing process at that time began with a fear of like each request may not be served by all the engines because of uh constraints such as capabilities or due uh restrictions due to compute or data residency. And among the remaining engines, IRB used a weighted consistent hashing to select the best destination engine for a request or of for certain user. Then the important question becomes where are the WS come from. So they were generated by a periodic feedback loop. The inference engines as mentioned earlier uh reports all kind of the signals we care about and the controller will periodically smooth out those signals and compute a performance score. The performance score then will be compared against the fleet average. Then the weight will be adjusted basically for each engine. is weight goes up if the performance is better or it goes down when the performance is worse than the fleet average and this generated weight will impact the routing and then it's basically a uh control loop um conceptually it is very similar to the P controller and no this P controller will not help you care a Linux process but instead it's a classic control theory technique that continuously steering the system towards its desired date and we just borrowed this important concept the proportional part of it and uh applied into our uh our load balancer. So it has a lot of nice properties. For example, it could combine the useful signals we care about into the single routing decision and because of the it adapt to the observed performance as what we mentioned earlier there's a lot of constraints and those constraints might have the some engines basier because they can serve more requests more kind of request than the remaining but those basier signal will be fit into the next loop and resulting in the less constraint less constraint request can go to more of those kind of engines. So basically they self balanced out and to some extent this just means we don't need to do a lot of manual intervention and it just works. However that kind of adaptability comes with big trade-offs because of the same reasons that it combined so many signals. It's also very hard to reason about a particular routing decision or like why search engine get a higher weight than we expect. And every time we want to fine-tune towards some aspect, it's all almost impossible to not impacting something else. And the load is not always very well very evenly distributed because uh sometimes a model is served by engines on different GPU skills and they have different characteristics. Then the problem becomes a lot more trickier and the feedback loop sometimes creates bad oscillations because when you shift an engine away some traffic the engine turns a bit cooler and this signal get fit to the controller. The controller now thinks that hey this engine can take a lot more traffic. then the some traffic going to be shifted back and forth between a few engines and disrupting the KV cache utilization. So all those limitations motivated us to rethink about the architecture and uh see if we have new ways to address the problem. So I'm going to hand over to Chenu to uh deep dive into the new architecture we tried out. \>\> Yeah, thank you L. So I'm going to talk about the architecture of the load balancer and how do we reduce the overall overhead with our routing algorithm. The load balancer answers one question uh for each request from a CPU cluster which engine should serve it. One most naive baseline might be round robin which send requests across engines evenly. But if you think a little bit more that doesn't make sense because engines are not homogeneous they can have different hardware and capacity different health and also different distance from CPU cluster also run could break cache locality c related requests that could reuse the same engine cache might be sent to different engines. A probably better solution might be for each CPU cluster it choose the best engine from its own local view. But that's not enough either. Think about one extreme case. Multiple CPU cluster route traffic to the same engines independently which could overload that engine while leave other engines underutilized. So what we need is a globally optimized solution, a control plane that has a global view for all the CPU cluster and GPU engines and could compute a globally optimized routing answers and the data plane can make a routing decision quickly based on the answer proved from the control plane. Now let's look inside the control plane and data plane. In the data plane there is an engine selector which select engine for each request. It read the local routing state which includes the candidate engines and the routing weights for each candidate engines. Both of them are refreshed asynchronously in the background. So we don't need to ask the control plane before we make a routing decision for each request. Also the data plane collects realtime engine signal such as number of ready replica engine house etc to surface fast local guard draw in the control plane. The data loader combines those live engine signals and never overhead. And with offline regressions of capacity, TTFT and TBOT, the optimizer could turn those data into routing weights and the control plane will publish the routing weight for each data plane to pull. In this way, no request need to wait on the data plane. The control plane continuously compute the next globally optimized routing way snapshot while the data plane make a routing decision based on the latest snapshot already installed locally. In summary, there are three important paths through the system. The first path is the inference request path. The request arrive to the CPU cluster and the data plane inside that CPU cluster will select engine for that request based on the local routing state and forward the request to the selected engines. The second pass is the engine signal pass. The system continuously collects real-time engine signal such as TTFT, TBOT, number of radio replica and engine house etc. Boost planes need those real-time engine signals. The control plane need them to compute a globally optimized routing way while the data plane need them to serve as fast local. And the third path is the routing way pass. The control plane compute and publish the routing way and the data plane pull the updates to its local cache. So only the first pass is synchronous but it's fast and only local inside the data plane of the CPU cluster. The other two loops are asynchronous loop and they are to improve future routing decision. So that's pretty much of the architecture part. But that still leaves one question. How do we compute those routing weights? But before answer that question, let's answer another question first. Why not just send a request to the nearest engine? That's because the traffic demand and GPU capacity are not geographically balanced. For example, in region one, CPU cluster A send 90 RPS and the nearby engine A can serve 100 RPS. So in this case nearest only is fine while in region two CPU cluster B send 120 RPS and the nearby engine B could only serve 100 RPS. So in this case if we insist on keeping everything local the extra 20 RPS need to wait on an overloaded engine B. While in region three we are only using 40 RPS of an 80 RPS engine C. That still leaves 40 RPS spare. So if we send the extra 20 RPS from cluster B to engine C, that will add network distance. But it could also avoid a probably much larger engine side waiting time. So in this case, a further engine might be faster end to end. That's why we need something better than the nearest only routing. Now let's open the black box of the optimizer. The optimizer accepts four types of input. The request from each CPU cluster, the network latency to each engine, the available engine capacity and health and also the TTFT, TBOT latency profiles that tell us how's the engine side latency change as the low increases. And with those input the optimizer turn the input to the output routing weights. The routing way say for each CPU cluster what fraction of its traffic should go to each GPU engine. And the optimization goal is straightforward is to minimize the expected end to end latency across all routed traffic. The important part is that the end to end latency includes both the network distance and the engine side latency. That means a nearby engine might be attractive when it still has room to serve traffic while a further engine might be better if all the nearby engines are close to full. And the optimizer also need to respect several hard constraints. First it need to route all the traffic demand. Second it need to ensure all the engines stay within the effected capacity. Third it need to keep the routing weights non- negative. With this the controller control plane get the routing way from the optimizer and publish them and the data plane pull them and use them to make a globally optimized routing decision. And that's pretty much of my part and Lou will continue to talk about the protection mechanisms in the system. Thanks Chenu. So as AI engineers we all kind of know that production in many cases are not behaving in the most ideal case. So clusters can fail, GPUs or individual nodes can degrade degree and networking can just get to all kind of mysterious issues. So how do we keep our production system uh heresy as much as possible under the heavy load? The first thing we have is the penalties. Basically when an engine is an outlier, we detect the try to reduce the routing weight to that engine. In that way, we give it a chance to either recover by themselves if there's a uh if it's some transient issue or we can have a human intervented out or replace the faulty hardware. And secondly, the retries which is a very common technique used to mitigate a problems. However, during some cases, it's actually could make them even worse like when the system is very close to like a tip over or very heavy utilized retries, we are send more load and this more load we are cause more failure and cause more retries which is infamous retry storm. So we incre implemented caps or budget to constant retries into a acceptable region and this is actually even need to be dynamic because in the happy time or in the normal time we can tolerate a lot more retries than when the system are heavily utilized. And finally we have the load shedding which is our last result when the production capac uh capacity couldn't meet the increasing amount of inference demands. So we instead try to have all the system fail. We basically proactively load shed a portion of the traffic to have the system degraded gracefully. So that pretty much concludes our talk today and uh thanks for joining us. Uh both of us will be around in our uh booth area this afternoon. So if you have further questions, feel free to walk uh to the area and and chat with us. Thank you.