--- schema: 1 type: video title: 'Homa: The End of TCP for AI Clusters — John Ousterhout, Stanford' description: >- John Ousterhout, Stanford Professor and author of A Philosophy of Software Design, turns our attention to the evolving nature of AI networking workloads and why traditional protocols like TCP and RDMA are becoming bottlenecks in modern data center environments! transcript/notes: https://ai.engineer/talks/eZ8WWZzoaR0-homa-end-tcp-ai-clusters Paper: https://www.usenix.org/system/files/atc21-ousterhout.pdf Related: https://lwn.net/Articles/1003059/ ## Shift in Workloads Historical context: AI traffic was dominated by massive, long-running transfers (gigabytes of gradients), where throughput was the primary metric (2:19-2:42). Modern AI: Workloads, especially inference and agentic applications, now rely on frequent, small coordination messages (e.g., KV cache lookups, barrier synchronization). These small messages are highly sensitive to latency (3:09-4:02). The Bottleneck: When small synchronization messages are mixed with large traffic, they get trapped in queues (caused by incast), significantly increasing 99th percentile (tail) latency. This causes GPUs to sit idle, wasting expensive compute resources (4:24-5:34). Why Legacy Protocols Struggle Sender-Driven Congestion Control: TCP and RDMA rely on the sender to detect congestion, often via packet drops or delayed signals from switches. This process is inherently reactive and oscillates, leading to unstable performance (7:10-9:58). Byte Stream Model: These protocols view data as an opaque stream of bytes rather than discrete messages, making it difficult to prioritize short, critical tasks (10:05-11:05). The Homa Solution John introduces Homa, a clean-slate transport protocol designed for data centers (11:15-12:15): Message-Based: Unlike byte streams, Homa understands message boundaries, allowing it to predict traffic and prioritize short messages using Shortest Remaining Processing Time (SRPT) (12:22-13:28). Receiver-Driven: The receiver controls the flow by issuing grants to senders, effectively managing congestion before it occurs at the switch (13:30-14:58). Priority Queues: Homa leverages the multiple hardware queues already present in modern switches to bypass long, queued traffic with low-latency short messages (14:59-15:51). Performance Results: Benchmarks show Homa can reduce tail latency for short messages by over 10x compared to TCP, while simultaneously improving performance for large messages (15:52-17:27). Speaker info: - https://x.com/johnousterhout - https://web.stanford.edu/~ouster/cgi-bin/home.php Timestamps: 0:00 - Why latency is becoming the metric that matters 2:19 - The old workload: gigabytes and throughput 3:09 - The new workload: metadata and coordination 4:24 - How one slow exchange stalls every GPU 6:07 - Incast, and where the queue actually builds 7:10 - Why congestion control lives on the wrong end 10:05 - A byte stream has no message boundaries 11:08 - Homa, and a clean slate redesign 12:12 - Messages, not streams 13:30 - Controlling congestion from the receiver 14:59 - Using the priority queues already in the switch 15:52 - The benchmark against TCP canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/homa-the-end-of-tcp-for-ai-clusters-john-ousterhout-1704d494b32b markdown: >- https://liquidgold.cc/channel/aidotengineer/video/homa-the-end-of-tcp-for-ai-clusters-john-ousterhout-1704d494b32b.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T03:36:48.748Z' language: null children: [] videoId: eZ8WWZzoaR0 durationSeconds: 1128 publishedAt: '2026-09-17T13:00:08.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 37 endSeconds: 149 watch: 'https://liquidgold.cc/watch/eZ8WWZzoaR0?t=37' heldReason: rights - ordinal: 1 text: null startSeconds: 196 endSeconds: 206 watch: 'https://liquidgold.cc/watch/eZ8WWZzoaR0?t=196' heldReason: rights - ordinal: 2 text: null startSeconds: 206 endSeconds: 272 watch: 'https://liquidgold.cc/watch/eZ8WWZzoaR0?t=206' heldReason: rights - ordinal: 3 text: null startSeconds: 282 endSeconds: 370 watch: 'https://liquidgold.cc/watch/eZ8WWZzoaR0?t=282' heldReason: rights - ordinal: 4 text: null startSeconds: 370 endSeconds: 382 watch: 'https://liquidgold.cc/watch/eZ8WWZzoaR0?t=370' heldReason: rights - ordinal: 5 text: null startSeconds: 448 endSeconds: 613 watch: 'https://liquidgold.cc/watch/eZ8WWZzoaR0?t=448' heldReason: rights - ordinal: 6 text: null startSeconds: 613 endSeconds: 624 watch: 'https://liquidgold.cc/watch/eZ8WWZzoaR0?t=613' heldReason: rights - ordinal: 7 text: null startSeconds: 741 endSeconds: 752 watch: 'https://liquidgold.cc/watch/eZ8WWZzoaR0?t=741' heldReason: rights - ordinal: 8 text: null startSeconds: 827 endSeconds: 903 watch: 'https://liquidgold.cc/watch/eZ8WWZzoaR0?t=827' heldReason: rights - ordinal: 9 text: null startSeconds: 903 endSeconds: 914 watch: 'https://liquidgold.cc/watch/eZ8WWZzoaR0?t=903' heldReason: rights - ordinal: 10 text: null startSeconds: 957 endSeconds: 969 watch: 'https://liquidgold.cc/watch/eZ8WWZzoaR0?t=957' heldReason: rights --- # Homa: The End of TCP for AI Clusters — John Ousterhout, Stanford John Ousterhout, Stanford Professor and author of A Philosophy of Software Design, turns our attention to the evolving nature of AI networking workloads and why traditional protocols like TCP and RDMA are becoming bottlenecks in modern data center environments! transcript/notes: https://ai.engineer/talks/eZ8WWZzoaR0-homa-end-tcp-ai-clusters Paper: https://www.usenix.org/system/files/atc21-ousterhout.pdf Related: https://lwn.net/Articles/1003059/ ## Shift in Workloads Historical context: AI traffic was dominated by massive, long-running transfers (gigabytes of gradients), where throughput was the primary metric (2:19-2:42). Modern AI: Workloads, especially inference and agentic applications, now rely on frequent, small coordination messages (e.g., KV cache lookups, barrier synchronization). These small messages are highly sensitive to latency (3:09-4:02). The Bottleneck: When small synchronization messages are mixed with large traffic, they get trapped in queues (caused by incast), significantly increasing 99th percentile (tail) latency. This causes GPUs to sit idle, wasting expensive compute resources (4:24-5:34). Why Legacy Protocols Struggle Sender-Driven Congestion Control: TCP and RDMA rely on the sender to detect congestion, often via packet drops or delayed signals from switches. This process is inherently reactive and oscillates, leading to unstable performance (7:10-9:58). Byte Stream Model: These protocols view data as an opaque stream of bytes rather than discrete messages, making it difficult to prioritize short, critical tasks (10:05-11:05). The Homa Solution John introduces Homa, a clean-slate transport protocol designed for data centers (11:15-12:15): Message-Based: Unlike byte streams, Homa understands message boundaries, allowing it to predict traffic and prioritize short messages using Shortest Remaining Processing Time (SRPT) (12:22-13:28). Receiver-Driven: The receiver controls the flow by issuing grants to senders, effectively managing congestion before it occurs at the switch (13:30-14:58). Priority Queues: Homa leverages the multiple hardware queues already present in modern switches to bypass long, queued traffic with low-latency short messages (14:59-15:51). Performance Results: Benchmarks show Homa can reduce tail latency for short messages by over 10x compared to TCP, while simultaneously improving performance for large messages (15:52-17:27). Speaker info: - https://x.com/johnousterhout - https://web.stanford.edu/~ouster/cgi-bin/home.php Timestamps: 0:00 - Why latency is becoming the metric that matters 2:19 - The old workload: gigabytes and throughput 3:09 - The new workload: metadata and coordination 4:24 - How one slow exchange stalls every GPU 6:07 - Incast, and where the queue actually builds 7:10 - Why congestion control lives on the wrong end 10:05 - A byte stream has no message boundaries 11:08 - Homa, and a clean slate redesign 12:12 - Messages, not streams 13:30 - Controlling congestion from the receiver 14:59 - Using the priority queues already in the switch 15:52 - The benchmark against TCP Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/homa-the-end-of-tcp-for-ai-clusters-john-ousterhout-1704d494b32b · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.