Talk by Stanford professor John Ousterhout on Homa, a clean-slate data center transport protocol designed for AI cluster networking. Covers how AI workloads have shifted from large throughput-heavy transfers to frequent, latency-sensitive small coordination messages (e.g., KV cache lookups, barrier synchronization), and how mixing these with large traffic causes incast-driven queueing that inflates tail latency and leaves GPUs idle. Explains why TCP and RDMA struggle: sender-driven congestion control reacts too late and oscillates, and the byte-stream abstraction lacks message boundaries for prioritization. Describes Homa's design—message-based semantics with SRPT prioritization, receiver-driven congestion control via grants, and use of hardware priority queues in switches—with benchmarks reporting over 10x reduction in tail latency for short messages versus TCP while improving large-message performance. Includes a full timestamped outline. # Channel Presentation Observations — "Homa: The End of TCP for AI Clusters — John Ousterhout, Stanford" ## Intended Audience The video addresses a **technical, practitioner-heavy audience** working in AI infrastructure and data center networking: - The speaker conducts an **audience poll of operators**: *"Is there anybody here where you have reason to believe that the latency of small messages is impacting the overall throughput of your applications? If so, can you just raise your hand?"* — and notes *"more hands than I expected,"* confirming practitioners are present. - Closing call-to-action targets the same group: *"as you're running your applications and measuring performance and seeing what the bottlenecks are, ask yourself, is high latency for short messages affecting your throughput?"* - Assumed background knowledge includes **TCP, RDMA/RoCE, ECN marking, incast, P99 tail latency, SRPT, kernel modules** — no introductory explanations are given, implying an expert audience. - Context clues (a live conference stage, *"fog generators in the auditorium," "really San Francisco experience"*) suggest an **in-person industry conference**, likely AI/infrastructure-focused. ## Claimed Scope The talk makes a **narrow, problem-focused claim**, not a broad content promise: - Explicit three-part agenda: *"First, to convince you that in fact the workloads are changing... Second... legacy protocols like TCP and RDMA are poorly suited to this environment... Then third, I'd like to introduce Homa."* - Domain scope: **network transport for AI clusters** — specifically *"the network side of AI applications"* and the shift from throughput-dominated to latency-sensitive workloads. - The scope is **persuasive/advocacy**, framed around one project: Homa, described as *"a new protocol we've developed at Stanford."* ## Self-Description Evidence The speaker offers explicit identity and role claims (creator statements, not demonstrated patterns): - **Affiliation**: introduced as *"Professor Emeritus at Stanford University, John Ousterhout."* - **Personal investment**: *"this is Homa is basically my life mission right now. I've sort of semi-retired from Stanford, and the reason I did that is so I can spend 100% of my time hacking on Homa."* - **Hands-on identity**: *"I'm not like most professors in that I love to code... I created a kernel module for Linux. I'm currently working through the process of getting that upstreamed... It's available on GitHub."* - **Origin story**: Homa *"started off as a PhD dissertation for one of my students, Ben Moazeni."* - **Direct engagement offer**: *"I'd be delighted to work with you and help you if you decide you want to experiment with Homa... feel free to contact me."* ## Demonstrated (vs. Claimed) Patterns - **Demonstrated**: deep technical exposition with concrete benchmarks (*"P99 for short messages... about 13 times faster"* than TCP) and mechanistic explanations of congestion control and SRPT prioritization. - **Claimed but not independently verified in the material**: performance figures rest on *"one sample benchmark that I use as part of my tuning and evaluation"* — the creator's own evaluation, presented by the protocol's author. - Note: this is a single conference talk; **no channel-wide patterns** (upload cadence, series structure, recurring formats) can be inferred from this material alone. # Delivery Observations: John Ousterhout — "Homa: The End of TCP for AI Clusters" ## Overall Tone and Register - **Conversational-academic**: The talk blends a professorial explainer style with casual asides, e.g., "Uh by the way, when I say RDMA, what I really mean is Rocky RDMA over converged Ethernet." - **Light humor, used sparingly**: Opens with situational wit — "I've never before given a talk where there are fog generators in the auditorium. Just a really San Francisco experience, I guess." - **Steady, low-key enthusiasm**: Emphasis comes through repetition rather than volume: "getting this right is really hard, really hard." ## Structure and Signposting - **Explicit numbered agenda**: "I hope to do three things over the next 15 or 20 minutes. First... The second thing... Then third..." - **Clear section transitions**: "So, let's dive in," "So, what do we do? Well, what I'd like to do next is tell you about a new protocol called Homa." - **Explicit wrap-up signal**: "So, just to wrap up..." - **Enumerated internals**: Within Homa, "I want to mention three things. First, it's message-based... The second thing about Homa... The third aspect of Homa..." ## Teaching Habits - **Worked examples with concrete numbers**: "three nodes can transfer three times as fast as one node can possibly receive"; "if the computation phase is, say, 5 seconds, and it takes a few milliseconds for the exchange, you know, not a problem." - **Rhetorical questions to anticipate audience doubts**: "Now, you you might wonder, why are senders doing it? Cuz the congestion is way over at the other end of the data center network." - **Definition-by-contrast**: TCP stream model explained via the slide — "there are no colors in real life. TCP has no idea where the message boundaries are." - **Graph walkthrough ritual**: Carefully names axes before interpreting — "on the x-axis is the message length... The y-axis shows you the round-trip time." - **Live audience poll**: "I'd like to just do a quick audience poll here... can you just raise your hand?" — followed by reaction: "Actually, more hands than I expected." - **Pre-empting a likely objection**: "you might think that because Homa favors shorter messages, that long messages suffer... It turns out that's actually not the case." - **Deferring depth**: "I don't have time to explain that today, but it has to do with..." ## Recurring Language Patterns - **Repetition for emphasis** (frequent): "it it seems that those workloads are changing"; "constantly oscillating between sending too too and and too little"; "really hard, really hard." - **Softening hedges**: "it appears that the workloads are changing," "this seems to be particularly true in the world of inference," "I think I think it's likely that it's going to continue to increase." - **Filler-heavy delivery**: frequent "So," "you know," "uh," and "and and" — consistent with an informal spoken register. - **"Really" as intensifier**: "really great networking performance," "really low latency," "the results were so great." ## Persuasion and Pitch Techniques - **Quantified claims**: "reduce tail latency by an order of magnitude or more"; "about 13 times faster." - **Problem-solution arc**: builds the case ("all in all, TCP and RDMA are just not well suited to this environment") before introducing the fix. - **Personal investment framing** at the close: "this is Homa is basically my life mission right now"; "I love to code"; offers direct help — "bug fixes, whatever, you know, I'd be happy to work with you." - **Direct call to action with low-friction channels**: "You should give Homa a try... My email's on the slide, or you can Google me, too." ## Uncertainty Notes - Single-talk sample; habits like the audience poll and graph walkthrough appear once each and may or may not be stable across talks. - Verbal fillers and repetition are consistently observable here but their frequency relative to other speakers cannot be established from one transcript. - Tone is inferred only from word choice and asides in the transcript; no vocal qualities (pace, pitch) are directly observable. # Position Record: Homa: The End of TCP for AI Clusters **Speaker:** John Ousterhout, Professor Emeritus, Stanford University ## Central Claim AI workloads are shifting from throughput-dominated large transfers toward latency-sensitive small messages, and legacy transports (TCP, RDMA/RoCE) handle these poorly due to tail latency under incast. Homa, a clean-slate transport protocol developed at Stanford, reduces tail latency by an order of magnitude or more. ## Positions and Reasoning ### 1. AI Workloads Are Changing - **Historical workloads:** enormous transfers (gigabytes, e.g., weight gradients) between machines; **throughput** is the only metric that matters. Connection setup overhead is negligible for long transfers; TCP and RDMA (i.e., RoCEv2, which he clarifies is the underlying transport) performed "pretty well." - **New workloads:** more granular, smaller exchanges — especially in **inference and agentic** workloads (less so training). Small messages carry metadata and coordination, e.g., KV cache lookups in a distributed cache, barrier synchronization. - **Tail latency matters, not just average:** if one straggler exchange delays a synchronization step, all nodes stall and GPUs sit idle. - **Aggravating trend:** with agentic workloads emitting tokens at a regular rate, compute phases shrink toward milliseconds; millisecond-scale synchronization then wastes a significant fraction of GPU resources. - **Stated uncertainty:** he expects the small-message trend to continue, but qualifies it — "We'll see over the next year or two if that happens." - **Audience poll:** he asked whether small-message latency affects attendees' throughput; he noted "more hands than I expected." ### 2. Why TCP and RDMA Have High Tail Latency **Cause: incast congestion** - Multiple nodes simultaneously send to one destination; packets queue at the last-hop egress port; short messages get stuck behind long ones. Worst case: buffer overflow, drops, timeouts, retransmissions. **Sender-side congestion control is fundamentally flawed** - Nearly all pre-Homa protocols (including TCP, RDMA) put congestion control at the sender, far from where congestion occurs. - Legacy detection via packet loss; modern improvement via ECN marking at switches relayed back via acknowledgements. - **Why it's "really hard":** senders get ~one bit of congestion information; multiple senders adjust simultaneously; **control lag** means several round trips to converge, by which time conditions have changed — leading to perpetual oscillation. - **His historical framing:** the problem has been known in the research community for 20+ years with many papers and some improvements, but "we're still a long ways from anything that works well" — because of the fundamental sender-side approach, not the specific algorithms. **Byte-stream data model is also a problem** - TCP serializes messages into a stream with no message boundaries; you can't know how much data is coming, can't prioritize short messages, and get **head-of-line blocking** (a short message stuck behind large ones in the stream). ### 3. Homa as the Solution - Origin: **Ben Moazeni's** PhD dissertation; Ousterhout then adopted it as a personal project ("my life mission right now"; he says he semi-retired from Stanford to work on it full time). Linux kernel module, being upstreamed, available on GitHub. **Three design differences:** 1. **Message-based, not stream-based** - Fundamental unit is a **remote procedure call** (request + response). - Message lengths are known to the transport, enabling prediction ("as soon as a receiver gets the first packet… it knows exactly how much more data is coming"), **SRPT** (shortest remaining processing time first) prioritization, and independent messages that bypass each other. 2. **Receiver-based congestion control** - Congestion occurs primarily at the last downlink to the receiver, so the receiver has near-complete information. - Mechanism: sender transmits only **unscheduled packets** initially; remaining **scheduled packets** are sent only upon **grant packets** from the receiver, which can pace grants to limit congestion and favor shorter messages. 3. **Uses switch priority queues** - Modern switches (typically ~8 egress queues) let packets specify queue priority; Homa dynamically assigns short messages to higher-priority queues so they bypass queued incast traffic. ### 4. Benchmark Claim - Workload of machines exchanging messages from ~50 bytes to 1 MB; measured P50 and P99 round-trip times, TCP vs. Homa. - **P99 for short messages:** TCP > 1 ms vs. Homa < 100 µs — ~13× faster. - **Long messages:** counterintuitively, Homa is also almost 2× better than TCP; he attributes this to **run-to-completion** scheduling versus TCP's fair scheduling, explicitly deferring explanation ("I don't have time to explain that today"). ## Call to Action - He asks practitioners to check whether short-message latency is limiting their throughput; if so, "give Homa a try" — expected order-of-magnitude tail-latency reduction. - Offers direct collaboration: help getting started, answering questions, bug fixes; contact via email on slides or via web search. ## Unresolved / Qualified Points - The projected continued growth of small-message workloads is explicitly a **prediction**, not settled ("I think it's likely… We'll see"). - The long-message performance advantage of Homa is asserted with a benchmark but not fully explained in the talk. - Tail-latency claims rest on **one sample benchmark** he uses for his own tuning and evaluation. # Speaker & Provenance Record — "Homa: The End of TCP for AI Clusters" ## Speakers & Roles - **John Ousterhout** — **Primary speaker**, identified in the transcript introduction as **Professor Emeritus at Stanford University**. He delivers the full talk. - Self-described role: developer/advocate of **Homa**, a transport protocol developed at Stanford; started as a PhD dissertation by his student **Ben Moazeni**, then became Ousterhout's "personal project." - States he is **"semi-retired from Stanford"** and spends "100% of my time hacking on Homa." - **Unidentified announcer** — Introduces the speaker ("Please welcome to the stage…"). No name given. - **Audience** — Responds to an informal hand-raising poll; no individual speakers identified. ## Attribution Notes - All technical claims (AI workload shifts, incast congestion, Homa's design, benchmark results) are **attributed solely to Ousterhout** in the talk; no co-authors or external reviewers speak. - Benchmark figures cited (e.g., Homa tail latency "<100 microseconds vs. TCP >1 millisecond, ~13x faster"; "almost a factor of two better" on long messages) are **speaker's own claims**, not independently verified in the transcript. ## Referenced Sources & Third-Party Claims - **TCP** and **RoCE (RDMA over Converged Ethernet)** — named as legacy protocols; speaker notes RoCE is "the underlying transport used by RDMA for most purposes today." - **ECN (early congestion notification)** — described as an existing congestion-signaling technique. - **SRPT (shortest remaining processing time first)** — cited as Homa's prioritization scheme. - Research community: speaker claims the congestion-control instability problem "has been known… for more than 20 years" with "tons of papers published" — no specific citations given. - **Homa** — claims to be **open source, available on GitHub**, with a Linux kernel module **in the process of being upstreamed** into the kernel (stated, not verified in transcript). ## Promotional Content — FLAGGED - The talk functions substantially as **promotion for Homa**: - Direct call to action: "You should give Homa a try." - Personal recruitment: Ousterhout offers help getting started, answering questions, and bug fixes; says his email is "on the slide." - His declared life mission and semi-retirement to work on Homa constitute a **disclosed personal stake** in adoption. ## Availability & Context Limits - **No comment availability information** is present in the transcript. - **No curated metadata supplied** beyond title and transcript. - Referenced but unavailable artifacts: **presentation slides** (benchmark graph, contact email), **GitHub repository**, and audience poll results — none independently verifiable from the transcript. \>\> Please welcome to the stage the Professor Emeritus at Stanford University, John Ousterhout. \>\> Good morning. It's really great to be here to talk about the network side of AI applications and in particular to make the case that latency matters and is probably going to be mattering more in the future. But I just want to say this is a talk is unusual for me. I've never before given a talk where there are fog generators in the auditorium. Just a really San Francisco experience, I guess. So it's it's well known that AI workloads depend on really great networking performance in order to achieve their own performance. Of course, that's because the workloads are so large that they have to be distributed across machines and then you have to communicate between the machines. But what I want to talk about today is it it seems that those workloads are changing. And so I hope to do three things over the next 15 or 20 minutes. First, to convince you that in fact the workloads are changing and that whereas the workloads used to be completely dominated by large transfers where throughput is the key metric that matters, that we're seeing more and more smaller transfers where the latency is crucial. The second thing I hope to do is to convince you that legacy protocols like TCP and RDMA are poorly suited to this environment. They weren't designed for this environment and unfortunately they suffer from very high tail latency when you mix small messages with large ones. And talk a little bit about why that's the case. Then third, I'd like to introduce Homa, which is a new protocol we've developed at Stanford that actually was designed in a clean slate redesign to handle data center workloads like these. And in fact, it does quite well on those workloads and can reduce tail latency by an order of magnitude or more. So, I'll tell you a little bit about Homa. So, let's dive in. First, workloads. Historically, AI workloads have consisted of enormous transfers between machines. And that's all that really mattered. Gigabytes of data for things like weight gradients and and so on. In these workloads, what you really care about is throughput. How many gigabits per second you can pump through the pipes. And these are relatively easy workloads for networks because if it takes a while to set up the connection and start the transfer, it doesn't matter. The transfers go on for so long that all that really matters is the throughput. And so, in these environments, TCP and RDMA perform pretty well. Uh by the way, when I say RDMA, what I really mean is Rocky RDMA over converged Ethernet, which is the underlying transport that's used by RDMA for most purposes today. So, anyhow, the old workloads, big transfers, throughput matters, uh the legacy protocols work pretty well. However, it appears that the workloads are changing. They're becoming more granular with smaller chunks of computation and smaller exchanges of data. And this seems to be particularly true in the world of inference and also in agentic workloads. Not so much for training workloads are still massive transfers. And so, what's happening is that more and more there are small message exchanges, typically for things like metadata and coordination, such as checking to see if a particular entry is present in a KV cache that's distributed, or doing barrier synchronization at the end of periods of compute. And for these workloads, what really matters is latency. That is, what's the round trip time to send some small piece of data across the network, do a little bit of computation, and get a small result back again? And in fact, it isn't just just latency or average latency that matters. What really matters is tail latency. That is, you'd like to know that if we send a whole lot of small messages, all of them will complete quickly. So, for example, we typically measure things like 99th percentile latency. And if we have high tail latency, that can limit the overall throughput of the system. So, here's an example. Suppose common thing is to take a workload and split it up across several nodes, which do intensive computation using their GPUs for some period of time. And then once they've all finished their computation, you do some small exchange between the nodes to exchange data and metadata, and then they'll go on to the next round of computation. And while that exchange is happening, that synchronization is happening, the GPUs are sitting idle. So, if even one of those exchanges takes a long time, it turns out the whole process stalls. You need all of those exchanges to complete before you can go on to the next phase of computation. Now, if the computation phase is, say, 5 seconds, and it takes a few milliseconds for the exchange, you know, not a not a problem. And that's historically what it's been. But now with the agentic workloads, where you're trying to pump out tokens relatively rapidly at a regular rate, the periods of computation are getting down into sort of the millisecond time scale. And if it also takes milliseconds to do that synchronization, then you're wasting a significant fraction of your GPU's resources waiting for the the synchronization to occur. So, I'm curious. I'd like to just do a quick audience poll here. Is there anybody here where you have reason to believe that the latency of small messages is impacting the overall throughput of your applications? If so, can you just raise your hand? See if there's anybody out there today. Actually, more hands than I expected. So, quite a few people out there are raising their hands. I think this problem is likely to get worse as the trends continue. So, what's going on? Why is tail latency bad? Well, typically the cause is congestion resulting from incast. So, incast is when several nodes all decide simultaneously to transfer data to some destination node. And if they all send large messages, well, the links are the same everywhere in the network. So, three nodes can transfer three times as fast as one node can possibly receive. And so, what happens is that packets accumulate at the last hop going to that destination in the top of your access switch at its egress port for the destination node. Then, if some other node decides it wants to send a short message to that same destination, the short message gets stuck behind the long ones in the queue there. And actually, as that causes delay, and in the worst case, so many packets arrive that the switch runs out of buffer space that it has to drop packets, and then there are there timeouts and retransmissions that make everything even worse. So, somehow we need some way to reduce the congestion in those queues. Somehow we have to get the sending nodes to stop sending so fast so the queues don't just build up without limit. So, the way this is done historically virtually all network protocols before Homa, including TCP and RDMA, congestion control is the responsibility of the sender. So, senders somehow have to figure out that congestion is happening, and then they have to slow down their rate of transmission. Now, you you might wonder, why are senders doing it? Cuz the congestion is way over at the other end of the data center network. How does the the sender find out? Well, in the in the old really old days, the way they would find out is the queues would overflow and packets would get dropped. The sender would detect the packets got lost cuz it wouldn't get acknowledgements back, and it would assume that means there's congestion and then slow down its rate of transfer. That's really expensive. So, today there are better techniques that mostly involve the switches providing information. So, a top of rack switch, when it sees that the queue length for an egress port has reached some threshold or starting to fill long before the queue overflows, it starts marking all of the packets to pass through with what's called early congestion notification, ECN marking. And so, when those packets pass through to the receiver, the receiver sees the marking in the packets, and then when it communicates back to the sender next, for example, to send an acknowledgement, then it includes that marking that goes back to the sender. And now the sender sees the sender realize, "Oh, there's congestion someplace. I've got to slow down my rate of transmission." So, that's the basic idea. Unfortunately, getting this right is really hard, really hard. It's very hard for the congestion to figure out exactly how to set its rates cuz it gets one bit of information. There's congestion someplace. And there are multiple senders all sending to the same destination. They're all trying to make adjustments simultaneously. How much do you cut back? And how do I know when I can ramp up again? And even worse, it's really hard to do this in a way that's stable because there's control lag. That is, it takes time before the sender finds out that there's congestion. And in fact, using this process, it typically takes several round trips for the sender to gradually adjust its rate to get just the right rate to match the available bandwidth. But by the time you do that, in a network that things have changed. New transmissions have started or old ones have finished. And so, these systems tend to never stabilize. They're constantly oscillating between sending too too and and too little. And this problem's been around for a long time. It's been known in the research community for more than 20 years now. There've been tons of papers published on it. There have been some improvements made, that's undeniable, but we're still a long ways from anything that works well. And the problem is with the fundamental nature of it doing the the congestion control at the sender side. It just doesn't work very well. So, you end up with a lot of queue build-up. And in fact, you can see the only way to find out that there's congestion is if there's queues. And so, by that point we're already experiencing delays. So, that's a problem. Oops. There's one other problem with TCP and RDMA also is that their their basic data model is a byte stream. Just a stream of bytes with no differentiation in it. So, if you send a series of messages say through a TCP socket, they get serialized into that stream. And on this slide, I've you know, I've shown the messages appear like they have different colors in the stream. Well, there are no colors in real life. TCP has no idea where the message boundaries are. And that also makes life hard. For example, you don't know how much more data is coming. If you knew how big the message was, you'd know how much more is coming. And you can't prioritize short messages, which we'd really like to do, get the short messages through faster. And you can end up with what's called head-of-line blocking, where somebody sends a series of messages to the same destination and they send two really large ones and then a small one after that, they get stuck behind them in that stream. And so, it gets delayed. And again, you have tail latency issues. So, all in all, TCP and RDMA are just not well suited to this environment. So, what do we do? Well, what I'd like to do next is tell you about a new protocol called Homa that we've developed at Stanford, which was based on a completely clean slate redesign for network transport. If you could start from scratch and rethink how you do transport for data centers, how would you do it? And it turns out in Homa, virtually every major design decision is different from TCP and RDMA. TCP for all the amazing things it's done is just not a good match to today's data centers, nor RDMA. So, what Homa does particularly well is to manage a combination of large and small messages and to make sure that the messages have rich short messages have really low latency. So, this started off as a PhD dissertation for one of my students, Ben Moazeni, and then the results were so great that I decided to make it my personal project to see if we can get it out of the lab and into production. Uh as you may know, I'm I'm not like most professors in that I love to code, and so I turned this into my my own programming project. I created a kernel module for Linux. I'm currently working through the process of getting that upstreamed into the kernel. It's available on GitHub for download. So, let me tell you just a little bit about how Homa works. I want to mention three things. First, it's message-based, not stream-based. \>\> \[clears throat\] \>\> In fact, the fundamental unit at Homa is a remote procedure call, which consists of two things: a request message sent from a client to a server, and then a response message returned back from the server to the client. So, the key thing here is that Homa knows about message lengths. They're buried in the transport all the way down to the bottom, and this has a bunch of advantages. First, it allows us to predict the future. As soon as a receiver gets the first packet of a message, it knows exactly how much more data the sender wants to send, and that's so much so much more information for doing congestion control. Second, Homa prioritizes shorter messages. It uses SRPT, shortest remaining processing time first, to try and prioritize shorter messages. And third, because messages are all independent, they're not serialized into a stream, every message is independent. Shorter messages can bypass long ones, so they don't get queued behind long messages. The second thing about Homa that's different is that it controls congestion from the receiver. Now, when you think about it, this makes sense because the congestion happens primarily at that last down link to the receiver. And so, the receiver has way more information. In fact, with Homa, as soon as it gets the first packet of a message, it knows exactly how much more is coming. So, it has essentially complete information about congestion and it can therefore respond to congestion much more quickly and much more precisely. The way things work with Homa is that when a sender has a message to send, it breaks it up into packets, but it only transmits the first few packets, those are called unscheduled packets, to the receiver. Packets after that are called scheduled packets, and they only get transmitted when the receiver asks for them. So, the receiver will send grant packets back. They'll paste them out and send those back to the sender over time, telling the sender, "It's now time for you to send me the next chunk of data." And the receiver can delay those grants. So, for example, if the receiver has 10 messages that are incoming, there's no point in sending grants to all 10 of them because then you'll just get congestion in the in the top of rack queues. So, we can use the grants to reduce congestion, and then it can also use the grants to give preference to its most favored messages, which would be the shorter ones. So, it's a way of of implementing SRPT by favoring short messages. The third aspect of Homa is that it takes advantage of the priority queues in modern switches. So, modern data center switches have more than one queue at each egress port, typically eight, and they can be used in a priority mechanism where packets get transmitted preferentially from the highest priority queue. So, I've shown only two queues on the slide here, but typically there's more than that. You can specify in a packets, using the various fields of the packet, you can specify which queue it should go into. And so, Homa dynamically makes those choices in a way to give priority to shorter messages. So, if we go back to the incast example from a few slides ago, all of those long messages will pile up in the lowest priority queue. But, if there's a short message coming, it will use a higher priority queue. And so, it will immediately bypass all of the queued packets from the short from the the longer messages and get through to the destination more quickly. So, how much of a difference does this make? Uh here's a uh this slide I've got one sample benchmark that I use as part of my tuning and evaluation of Homa. It consists of a workload of a bunch of machines on a network that are exchanging messages back and forth of different sizes, ranging from very small to very large. And on this graph, you can see on the x-axis is the message length, so from about 50 bytes up to a megabyte. The y-axis shows you the round-trip time for messages of that length. So, this request this uses request and response messages that are the same length. You can see TCP in green, Homa in blue. And the y-axis is is round-trip time, so lower is better. And for each protocol, I've got two curves. One curve is the P50 curve, that's the median latency for messages of this length. And then P99 is the 99th percentile, i.e. tail latency for messages of this length. So, I want to point out two things. First, the P99 for short messages is dramatically better for Homa. So, with TCP, it's more than a millisecond tail latency, Homa is less than 100 microseconds, about 13 times faster. Second, interestingly, you might think that because Homa favors shorter messages, that long messages suffer and get worse performance. It turns out that's actually not the case. Even on the longest messages, Homa is almost a factor of two better than TCP. I don't have time to explain that today, but it has to do with the fact that Homa uses run-to-completion approaches, which are much more effective than fair than the fair scheduling used by TCP. So, just to wrap up, the role of short messages in AI appears to be increasing. I think I think it's likely that it's going to continue to increase. We'll see over the next year or two if that happens. And I just want to pose a question to you, you know, as you're running your applications and measuring performance and seeing what the bottlenecks are, ask yourself, is high latency for short messages affecting your throughput? If the answer is yes, then just know there is a solution available. You should give Homa a try. You can probably reduce your tail latency by an order of magnitude or more. And by the way, this is Homa is basically my life mission right now. I've sort of semi-retired from Stanford, and the reason I did that is so I can spend 100% of my time hacking on Homa. So, I'd be delighted to work with you and help you if you decide you want to experiment with Homa. If you need help getting started, answer questions, bug fixes, whatever, you know, I'd be happy to work with you to try and make you successful with it. So, if that is interesting, feel free to contact me. My email's on the slide, or you can Google me, too, and find me over the internet. So, thanks very much for listening, and I hope to hear from some of you. \>\> \[music\]