--- schema: 1 type: video title: >- KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat description: >- Agentic sessions in Red Hat's traces run from a few turns to 3,000, cache hit rates routinely clear 90%, and input to output token ratios often pass 100 to 1. A public inference benchmark shows none of that, because it reports steady state numbers from one sanitized run. Yuchen Fama and Ashish Kamra spend the talk on the two levers that matter once the client rather than the server controls the cache lifecycle, and a live demo makes the first one concrete. An opening request takes about 3 seconds, the next turn reuses the cache on the same pod and takes about 1, and a fresh system prompt lands on a different pod and pays the full 3 again. With a 10x gap between cached and uncached token costs, routing is the cheaper lever to reach for before adding GPUs. The second lever splits compute bound prefill from memory bound decode, so a long incoming prompt cannot stall token generation midstream. Across 16 H100s serving gpt-oss, P99 inter token latency falls from roughly 900 milliseconds to about 100, and the curve gets visibly smoother. The useful part is where they draw the boundary. Disaggregation wins in the middle concurrency band, roughly ties at both ends, and needs an RDMA or RoCE fabric to move cache between workers at all. Without one, stay aggregated. The closing case study runs GLM 5.2 on the H200s customers actually have instead of B200s, at three prefill workers to one decode, for 4x faster time to first token and 60% more requests. Speaker info: - https://www.linkedin.com/in/yuchen-fama - https://www.linkedin.com/in/ashishkamra/ - https://github.com/llm-d/llm-d Timestamps: 0:00 - What public inference benchmarks leave out 1:28 - Red Hat's inference stack, and the agenda 3:29 - Agentic traces: 3,000 turns, 90% cache hits, 100 to 1 ratios 5:12 - Volatile cache, and the 10x cached token gap 6:28 - How llm-d routes: endpoint picker, offload tiers, eviction 7:46 - Demo: cache hits, pod affinity, 3 seconds versus 1 9:28 - What llm-d is, and why prefill and decode interfere 12:03 - How disaggregation works in practice 13:08 - P99 inter token latency: 900ms versus 100ms 14:14 - Where PD shines across the concurrency curve 15:45 - When to use PD, and when to stay aggregated 18:10 - GLM 5.2 on H200s: 4x faster TTFT --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/YXowceUKYJI-youtube-100be9dcc20eaeb3394e Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/YXowceUKYJI' markdown: 'https://liquidgold.cc/watch/YXowceUKYJI.md' llms: 'https://liquidgold.cc/watch/YXowceUKYJI/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T04:35:18.680Z' language: null children: [] videoId: YXowceUKYJI durationSeconds: 1308 publishedAt: '2026-08-27T14:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 24 endSeconds: 88 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=24' heldReason: rights - ordinal: 1 text: null startSeconds: 226 endSeconds: 314 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=226' heldReason: rights - ordinal: 2 text: null startSeconds: 226 endSeconds: 314 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=226' heldReason: rights - ordinal: 3 text: null startSeconds: 326 endSeconds: 391 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=326' heldReason: rights - ordinal: 4 text: null startSeconds: 401 endSeconds: 472 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=401' heldReason: rights - ordinal: 5 text: null startSeconds: 401 endSeconds: 472 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=401' heldReason: rights - ordinal: 6 text: null startSeconds: 482 endSeconds: 571 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=482' heldReason: rights - ordinal: 7 text: null startSeconds: 583 endSeconds: 723 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=583' heldReason: rights - ordinal: 8 text: null startSeconds: 723 endSeconds: 735 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=723' heldReason: rights - ordinal: 9 text: null startSeconds: 802 endSeconds: 858 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=802' heldReason: rights - ordinal: 10 text: null startSeconds: 869 endSeconds: 953 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=869' heldReason: rights - ordinal: 11 text: null startSeconds: 964 endSeconds: 1090 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=964' heldReason: rights - ordinal: 12 text: null startSeconds: 964 endSeconds: 1090 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=964' heldReason: rights - ordinal: 13 text: null startSeconds: 1101 endSeconds: 1308 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=1101' heldReason: rights - ordinal: 14 text: null startSeconds: 1101 endSeconds: 1308 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=1101' heldReason: rights - ordinal: 15 text: null startSeconds: 1101 endSeconds: 1308 watch: 'https://liquidgold.cc/watch/YXowceUKYJI?t=1101' heldReason: rights --- # KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat Agentic sessions in Red Hat's traces run from a few turns to 3,000, cache hit rates routinely clear 90%, and input to output token ratios often pass 100 to 1. A public inference benchmark shows none of that, because it reports steady state numbers from one sanitized run. Yuchen Fama and Ashish Kamra spend the talk on the two levers that matter once the client rather than the server controls the cache lifecycle, and a live demo makes the first one concrete. An opening request takes about 3 seconds, the next turn reuses the cache on the same pod and takes about 1, and a fresh system prompt lands on a different pod and pays the full 3 again. With a 10x gap between cached and uncached token costs, routing is the cheaper lever to reach for before adding GPUs. The second lever splits compute bound prefill from memory bound decode, so a long incoming prompt cannot stall token generation midstream. Across 16 H100s serving gpt-oss, P99 inter token latency falls from roughly 900 milliseconds to about 100, and the curve gets visibly smoother. The useful part is where they draw the boundary. Disaggregation wins in the middle concurrency band, roughly ties at both ends, and needs an RDMA or RoCE fabric to move cache between workers at all. Without one, stay aggregated. The closing case study runs GLM 5.2 on the H200s customers actually have instead of B200s, at three prefill workers to one decode, for 4x faster time to first token and 60% more requests. Speaker info: - https://www.linkedin.com/in/yuchen-fama - https://www.linkedin.com/in/ashishkamra/ - https://github.com/llm-d/llm-d Timestamps: 0:00 - What public inference benchmarks leave out 1:28 - Red Hat's inference stack, and the agenda 3:29 - Agentic traces: 3,000 turns, 90% cache hits, 100 to 1 ratios 5:12 - Volatile cache, and the 10x cached token gap 6:28 - How llm-d routes: endpoint picker, offload tiers, eviction 7:46 - Demo: cache hits, pod affinity, 3 seconds versus 1 9:28 - What llm-d is, and why prefill and decode interfere 12:03 - How disaggregation works in practice 13:08 - P99 inter token latency: 900ms versus 100ms 14:14 - Where PD shines across the concurrency curve 15:45 - When to use PD, and when to stay aggregated 18:10 - GLM 5.2 on H200s: 4x faster TTFT --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/YXowceUKYJI-youtube-100be9dcc20eaeb3394e Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/YXowceUKYJI · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.