--- schema: 1 type: video title: >- Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave description: >- On an agentic request, somewhere between 80 and 90 percent of the input is identical to the request before it, and prefill is the most expensive thing an inference stack does. That fact, Sitanshu Gupta argues, is why a cached input token is priced so far below a fresh one, and it shapes nearly every choice in the platform he now runs at CoreWeave, four months in. It has to serve two consumption models without forking. Serverless, where you pay per token and never see hardware, with a provisioned throughput tier for customers who know their traffic profile and would rather not share a noisy neighbor. And dedicated, billed per GPU hour behind a private gateway, where the customer picks the engine, decides whether prefill and decode are disaggregated, and owns the deployment. Underneath sit four workload shapes fitted together across time like Tetris. Agentic and chat look alike on paper, long inputs and short outputs, but agentic turns arrive with no human reading in between, so the latency budget is tiny. Voice and video are streaming and latency bound. Batch customers hand over twelve hours of work and say process it whenever, which is why dedicated capacity serving real time traffic by day can be scheduled to drain a batch queue overnight. The router prioritizes KV cache locality first and least loaded second, across heterogeneous GPUs in different zones, and between chat turns the cache is offloaded to high bandwidth storage rather than evicted, so the next question does not pay for prefill again. The two biggest levers lately are four bit quantization and speculators trained asynchronously on a customer's own data for better acceptance lengths. Speaker info: - https://www.linkedin.com/in/sitanshugupta Timestamps: 0:00 - Four months in, leading inference at CoreWeave 1:22 - Serverless, dedicated, and provisioned throughput 3:10 - Four workload shapes and a game of Tetris 5:15 - The request flow, from gateway to GPU 7:17 - Why the router is KV cache aware 10:21 - Scheduling batch into idle overnight capacity 11:05 - Offloading KV cache between turns 12:31 - Quantization and custom trained speculators --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/cQQbJqvZkpo-youtube-12b617db37296a8e176e Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/cQQbJqvZkpo' markdown: 'https://liquidgold.cc/watch/cQQbJqvZkpo.md' llms: 'https://liquidgold.cc/watch/cQQbJqvZkpo/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:21:06.570Z' language: null children: [] videoId: cQQbJqvZkpo durationSeconds: 922 publishedAt: '2026-09-19T16:30:34.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 24 endSeconds: 92 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=24' heldReason: rights - ordinal: 1 text: null startSeconds: 92 endSeconds: 102 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=92' heldReason: rights - ordinal: 2 text: null startSeconds: 92 endSeconds: 102 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=92' heldReason: rights - ordinal: 3 text: null startSeconds: 102 endSeconds: 200 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=102' heldReason: rights - ordinal: 4 text: null startSeconds: 211 endSeconds: 315 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=211' heldReason: rights - ordinal: 5 text: null startSeconds: 211 endSeconds: 315 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=211' heldReason: rights - ordinal: 6 text: null startSeconds: 315 endSeconds: 326 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=315' heldReason: rights - ordinal: 7 text: null startSeconds: 326 endSeconds: 447 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=326' heldReason: rights - ordinal: 8 text: null startSeconds: 447 endSeconds: 457 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=447' heldReason: rights - ordinal: 9 text: null startSeconds: 457 endSeconds: 621 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=457' heldReason: rights - ordinal: 10 text: null startSeconds: 457 endSeconds: 621 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=457' heldReason: rights - ordinal: 11 text: null startSeconds: 632 endSeconds: 665 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=632' heldReason: rights - ordinal: 12 text: null startSeconds: 677 endSeconds: 756 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=677' heldReason: rights - ordinal: 13 text: null startSeconds: 677 endSeconds: 756 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=677' heldReason: rights - ordinal: 14 text: null startSeconds: 769 endSeconds: 922 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=769' heldReason: rights - ordinal: 15 text: null startSeconds: 769 endSeconds: 922 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=769' heldReason: rights - ordinal: 16 text: null startSeconds: 769 endSeconds: 922 watch: 'https://liquidgold.cc/watch/cQQbJqvZkpo?t=769' heldReason: rights --- # Vertical Mobility: Inference from MVP to Trillion-Parameter Workloads — Sitanshu Gupta, CoreWeave On an agentic request, somewhere between 80 and 90 percent of the input is identical to the request before it, and prefill is the most expensive thing an inference stack does. That fact, Sitanshu Gupta argues, is why a cached input token is priced so far below a fresh one, and it shapes nearly every choice in the platform he now runs at CoreWeave, four months in. It has to serve two consumption models without forking. Serverless, where you pay per token and never see hardware, with a provisioned throughput tier for customers who know their traffic profile and would rather not share a noisy neighbor. And dedicated, billed per GPU hour behind a private gateway, where the customer picks the engine, decides whether prefill and decode are disaggregated, and owns the deployment. Underneath sit four workload shapes fitted together across time like Tetris. Agentic and chat look alike on paper, long inputs and short outputs, but agentic turns arrive with no human reading in between, so the latency budget is tiny. Voice and video are streaming and latency bound. Batch customers hand over twelve hours of work and say process it whenever, which is why dedicated capacity serving real time traffic by day can be scheduled to drain a batch queue overnight. The router prioritizes KV cache locality first and least loaded second, across heterogeneous GPUs in different zones, and between chat turns the cache is offloaded to high bandwidth storage rather than evicted, so the next question does not pay for prefill again. The two biggest levers lately are four bit quantization and speculators trained asynchronously on a customer's own data for better acceptance lengths. Speaker info: - https://www.linkedin.com/in/sitanshugupta Timestamps: 0:00 - Four months in, leading inference at CoreWeave 1:22 - Serverless, dedicated, and provisioned throughput 3:10 - Four workload shapes and a game of Tetris 5:15 - The request flow, from gateway to GPU 7:17 - Why the router is KV cache aware 10:21 - Scheduling batch into idle overnight capacity 11:05 - Offloading KV cache between turns 12:31 - Quantization and custom trained speculators --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/cQQbJqvZkpo-youtube-12b617db37296a8e176e Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/cQQbJqvZkpo · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.