--- schema: 1 type: video title: >- Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI description: >- The routing weights inside OpenAI's inference load balancer used to come out of a feedback loop. Engines reported signals, a controller smoothed them into a score, compared it to the fleet average, and nudged each weight up or down. A proportional controller, Lu Zhang notes, with real virtues: many signals folded into one decision, and constrained engines balanced themselves. It also produced behavior nobody could explain well. Ask why one engine got a higher weight and there was no clean answer. Tune one property and another moved. Worst was the oscillation: shift traffic off a hot engine, it cools, the controller reads cool as spare capacity and sends the traffic back, and the bouncing wrecks the KV cache locality routing was meant to protect. Qianru Lao walks through what replaced it: a control plane with a global view of every CPU cluster and GPU engine, and a data plane in each cluster that answers the one synchronous question, which engine serves this request, from a cached snapshot of routing weights. Signals still flow, into an optimizer rather than a loop. Its goal is to minimize expected end to end latency across all traffic, counting network distance and engine side queueing, under hard constraints that every request is routed and no engine exceeds capacity. Her example of why nearest is not enough: one region sends 120 requests a second at an engine that serves 100, while an engine two regions away sits at 40 of 80, so the farther engine wins once you count the wait. Zhang closes with the protections: outlier penalties, retry budgets that tighten as utilization climbs to prevent retry storms, and load shedding as last resort. Speaker info: - https://linkedin.com/in/qianru-lao - https://openai.com - https://www.linkedin.com/in/luzhang1/ Timestamps: 0:00 - From engine signal feedback loops to explicit policy 2:44 - What makes inference routing different 3:41 - Early days: weighted consistent hashing 4:23 - Weights from a proportional controller 6:57 - Oscillation that disrupts the cache 7:38 - Control plane, data plane, and a global view 9:18 - Three paths: request, signal, and routing weight 12:03 - Why not the nearest engine 13:28 - Inside the optimizer 15:38 - Penalties, retry budgets, and load shedding --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/sOB3HSiG8vo-youtube-f4b9f3d03417a2f74f1b Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/routing-llm-inference-in-production-from-engine-sig-e7b54c65aaf0 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/routing-llm-inference-in-production-from-engine-sig-e7b54c65aaf0.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:21:06.879Z' language: null children: [] videoId: sOB3HSiG8vo durationSeconds: 1092 publishedAt: '2026-09-19T15:30:04.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 23 endSeconds: 169 watch: 'https://liquidgold.cc/watch/sOB3HSiG8vo?t=23' heldReason: rights - ordinal: 1 text: null startSeconds: 180 endSeconds: 223 watch: 'https://liquidgold.cc/watch/sOB3HSiG8vo?t=180' heldReason: rights - ordinal: 2 text: null startSeconds: 280 endSeconds: 422 watch: 'https://liquidgold.cc/watch/sOB3HSiG8vo?t=280' heldReason: rights - ordinal: 3 text: null startSeconds: 280 endSeconds: 422 watch: 'https://liquidgold.cc/watch/sOB3HSiG8vo?t=280' heldReason: rights - ordinal: 4 text: null startSeconds: 434 endSeconds: 466 watch: 'https://liquidgold.cc/watch/sOB3HSiG8vo?t=434' heldReason: rights - ordinal: 5 text: null startSeconds: 480 endSeconds: 558 watch: 'https://liquidgold.cc/watch/sOB3HSiG8vo?t=480' heldReason: rights - ordinal: 6 text: null startSeconds: 740 endSeconds: 808 watch: 'https://liquidgold.cc/watch/sOB3HSiG8vo?t=740' heldReason: rights - ordinal: 7 text: null startSeconds: 820 endSeconds: 938 watch: 'https://liquidgold.cc/watch/sOB3HSiG8vo?t=820' heldReason: rights - ordinal: 8 text: null startSeconds: 820 endSeconds: 938 watch: 'https://liquidgold.cc/watch/sOB3HSiG8vo?t=820' heldReason: rights - ordinal: 9 text: null startSeconds: 948 endSeconds: 1092 watch: 'https://liquidgold.cc/watch/sOB3HSiG8vo?t=948' heldReason: rights - ordinal: 10 text: null startSeconds: 948 endSeconds: 1092 watch: 'https://liquidgold.cc/watch/sOB3HSiG8vo?t=948' heldReason: rights - ordinal: 11 text: null startSeconds: 948 endSeconds: 1092 watch: 'https://liquidgold.cc/watch/sOB3HSiG8vo?t=948' heldReason: rights --- # Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI The routing weights inside OpenAI's inference load balancer used to come out of a feedback loop. Engines reported signals, a controller smoothed them into a score, compared it to the fleet average, and nudged each weight up or down. A proportional controller, Lu Zhang notes, with real virtues: many signals folded into one decision, and constrained engines balanced themselves. It also produced behavior nobody could explain well. Ask why one engine got a higher weight and there was no clean answer. Tune one property and another moved. Worst was the oscillation: shift traffic off a hot engine, it cools, the controller reads cool as spare capacity and sends the traffic back, and the bouncing wrecks the KV cache locality routing was meant to protect. Qianru Lao walks through what replaced it: a control plane with a global view of every CPU cluster and GPU engine, and a data plane in each cluster that answers the one synchronous question, which engine serves this request, from a cached snapshot of routing weights. Signals still flow, into an optimizer rather than a loop. Its goal is to minimize expected end to end latency across all traffic, counting network distance and engine side queueing, under hard constraints that every request is routed and no engine exceeds capacity. Her example of why nearest is not enough: one region sends 120 requests a second at an engine that serves 100, while an engine two regions away sits at 40 of 80, so the farther engine wins once you count the wait. Zhang closes with the protections: outlier penalties, retry budgets that tighten as utilization climbs to prevent retry storms, and load shedding as last resort. Speaker info: - https://linkedin.com/in/qianru-lao - https://openai.com - https://www.linkedin.com/in/luzhang1/ Timestamps: 0:00 - From engine signal feedback loops to explicit policy 2:44 - What makes inference routing different 3:41 - Early days: weighted consistent hashing 4:23 - Weights from a proportional controller 6:57 - Oscillation that disrupts the cache 7:38 - Control plane, data plane, and a global view 9:18 - Three paths: request, signal, and routing weight 12:03 - Why not the nearest engine 13:28 - Inside the optimizer 15:38 - Penalties, retry budgets, and load shedding --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/sOB3HSiG8vo-youtube-f4b9f3d03417a2f74f1b Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/routing-llm-inference-in-production-from-engine-sig-e7b54c65aaf0 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.