Talk by Max Ryabinin of Together AI on training long-context LLMs, scaling a LLaMA-based model from 3 million to 5 million token contexts. Covers a memory-efficient training stack — fully sharded data parallelism, DeepSpeed Ulysses context parallelism, activation checkpointing, CPU offloading, and chunked sequence training — and introduces Untied Ulysses, an improvement to context parallelism that chunks attention head groups and reuses buffers to cut activation memory with negligible throughput impact. Reported results at 8B and 32B scale match memory-optimized training baselines while extending sequence length 25% beyond prior Ulysses implementations. # Video Identity Observations: "Road to 5 Million Tokens" — Max Ryabinin, Together AI ## Speaker & Affiliation - **Max Ryabinin**, who introduces himself as **"VP of research and development at Together AI."** - He explicitly attributes the research project to the organization: *"our research project, which is called Road to 5 million sequence length, breaking memory barriers in context parallelism."* ## Explicit Self-Description (Creator Claims) The speaker gives a direct organizational self-description of Together AI early in the talk: > "Together AI is an AI native cloud, which provides services and infrastructure for AI developers and builders at all stages of development." Claimed scope of services includes: - **Infrastructure**: GPU clusters "with heavily optimized computes and highly reliable systems" for model creation. - **Model shaping**: fine-tuning and reinforcement learning to customize models for "performance... speed... quality." - **Inference**: "the fastest way to launch and use AI models with more than 200 models in our portfolio," with serverless and dedicated deployment options. **Note**: These are *claims by the creator* about the company, not independently verified in the material. ## Intended Audience - **AI/ML engineers and researchers**, specifically those interested in **long-context model training** — evidenced by references to *FSDP, DeepSpeed Ulysses, Flash Attention, activation checkpointing, PyTorch profiler*, and Hugging Face training blog posts as familiar touchpoints. - The audience is framed as practitioners: *"I encourage you to use as well"* (recommending context parallelism), and acknowledgments that techniques are *"pretty well known to some in the community."* - The talk appears delivered at a **conference or meetup** format (closing with *"we're ready for questions"* and a live Q&A with audience questions about transformer internals). ## Claimed Research Scope - The talk claims a concrete, quantified research goal: training transformer models to **5 million token context lengths** on a single 8×H100 node. - Original contribution claimed: an optimization **"dubbed the Untitled Ulysses"** (later referred to as "U-Pipe") — chunked head-wise attention computation with reusable buffers, described as *"the primary optimization we've done in our work."* - Claims of results: *"both at the 8 billion scale and at the 32 billion scale, we're matching quite closely the most memory optimized implementations... while being able to scale even further like 5 million tokens."* - Public artifact claims: *"check out our paper for more results. All of that is public at the moment and we have an upcoming thread which will illustrate the method in more depth."* ## Demonstrated Presentation Patterns (Observable, Not Claimed) - **Technical depth**: Full walkthrough of a memory-optimization progression (parameters → FSDP → Ulysses context parallelism → activation checkpointing → CPU offloading → Arctic sequence-length tiling → U-Pipe), with specific numbers (e.g., ~8x utilization drops, ~37 GB offloaded data, 3B Llama architecture). - **Crediting prior work**: Attributes DeepSpeed Ulysses to Microsoft and CPU offloading to Unsloth — demonstrating a presentation style that situates own work within existing research. - **Problem-motivation framing**: Opens with community-driven motivations (agent applications, video generation requiring temporal consistency) rather than purely internal goals. - **Educational posture**: Includes a listener-oriented summary takeaway: *"training models with large context lengths is a very interesting and challenging goal, but the bottlenecks might appear where you least expect."* - **Narrow scope discipline**: Explicitly excludes parts of the company's portfolio from the talk: advanced inference optimizations *"which I will not be able to speak about today"*; the talk is *"focused on model training, customization, and fine-tuning."* ## Channel Identity Signals (for later synthesis) - The video is a **research-talk format** by a named corporate research leader, combining company marketing (services overview) with a technical research presentation and Q&A. - Self-positioning: a research organization publishing public papers and threads, targeting a technical ML engineering audience. - The speaker's closing aside — *"we're together and I am employees"* (apparently a response to an audience question about affiliation) — confirms the employee/spokesperson role, though the fragment is garbled and should not be over-interpreted. # Delivery Observations: Max Ryabinin — "Road to 5 Million Tokens" ## Tone & Register - **Conversational-academic tone**: blends technical depth with casual hedging phrases like "uh I think," "one might say," and "long story short." - **Self-deprecating aside in Q&A**: "cuz we're together and I am employees" (garbled transcript, but reads as an informal joke about colleagues attending). - Consistently welcoming and encouraging toward the audience: > "I encourage you to use as well" — recurring audience-inclusive framing. ## Vocabulary & Technical Framing - **Heavy systems/ML jargon delivered as familiar shorthand**: FSDP ("fully sharded data parallelism"), "DeepSpeed Ulysses," "activation checkpointing," "Flash Attention 1 2 3 4," "offload them to CPU," "PyTorch profiler." - **Memory-as-budget metaphor**: memory is something you "reinvest": > "maybe you might be able to reinvest it in some other ways and speed up your training overall" and later > "you could, for example, free up a bit of additional memory in your training if you need it and reinvest it somewhere else." - Attribution habit — credits prior work by name: > "first uh introduced by Microsoft," > "to the best of our knowledge, was first implemented uh by Unsloth." ## Teaching Habits & Structure - **Stepwise, incremental walkthrough**: presents optimizations as a ladder ("even with these optimizations, you are finally getting to a point where 3 million is possible. But, what if you wanted to go further?") — a repeated cliffhanger-style transition between stages. - **Numbered enumeration for motivation**: "the primary reasons for that are twofold" followed by "First of all... Second..." - **Rhetorical questions as pivots**: > "So, what happens next is..." (used at least twice); > "But, what if you wanted to go further?" - **Quantified claims with approximate framing**: "approximately 8x," "a further factor of eight," "next to 37 uh gigabytes of data," "more than 200 models." ## Recurring Phrases & Verbal Tics - **"Pretty" as an intensifier**, very frequent: "pretty well known," "pretty considerably," "pretty much all of the deep learning frameworks," "pretty difficult to deal with," "pretty significant." - **"Quite" as a hedge/intensifier**: "quite effective," "quite a few tokens," "quite straightforward," "quite insightful," "quite importantly." - **Fillers "uh" and "um" throughout**, plus filler "like" in looser passages (denser in the Q&A: "like allocated that whole big tensor which has 3 million..."). ## Communication Patterns - **Honest epistemic marking**: flags uncertainty explicitly — "to the best of our knowledge, was first implemented uh by Unsloth." - **Deferential scoping of the talk**: > "a ton of advanced optimizations, which I will not be able to speak about today." - **Closing redirect to written work**, repeated: > "check out our paper for more results. All of that is public at the moment" — the "public" emphasis appears twice. - **Q&A style**: clarifies misunderstanding patiently and concretely ("Uh not exactly. It was just the query, key, and value matrices..."), restating the core idea rather than just correcting. ## Uncertainty Notes - The transcript appears auto-generated with garbled segments (e.g., "3 by 5 million context lengths," "we elaborate on on in our paper"), so some phrasing patterns may reflect transcription noise rather than actual delivery. - Tone claims beyond verbal patterns (e.g., demeanor, pacing, gestures) cannot be assessed from transcript alone. # Observation Record: Road to 5 Million Tokens — Max Ryabinin, Together AI ## Speaker Identification - **Max Ryabinin**, VP of Research and Development at Together AI (per transcript self-introduction and curated title). ## Context: About Together AI Ryabinin describes Together AI as an **"AI native cloud"** offering: - GPU clusters for model creation - **Model shaping** via fine-tuning and reinforcement learning - **Inference services** with 200+ models, serverless and dedicated deployment options, and advanced optimizations (not detailed in this talk) ## Motivation for Long-Context Training Two stated drivers: 1. **Agents**: applications wanting to place as many tokens as possible into context and leverage it effectively 2. **Video generation**: multiple frames (even multiple per second) consume context quickly; models need **temporal consistency** — seeing what happened seconds to minutes prior He adds a secondary point: even at sub-million-token scales, understanding memory usage matters because freed memory could be reinvested to speed up training. ## Problem Statement Standard transformers scaling context hit two bottlenecks: - **Quadratic computation** from pairwise interactions across all sequence elements - **Linearly growing memory**, which he calls "more insidious" and difficult without specific techniques He cites a Hugging Face blog post example on sequence-length memory growth (referenced, not quoted). ## Progression of Techniques (Llama 3B, 3M train tokens, 8x H100 node) 1. **Baseline**: model parameters alone exceed GPU memory 2. **Fully sharded data parallelism (FSDP)**: parameters chunked across 8 GPUs; still OOM due to **attention activations** 3. **DeepSpeed Ulysses context parallelism** (first introduced by Microsoft): compute attention for different heads on different GPUs via communicated activations; one GPU responsible for one head over the whole sequence; compatible with Flash Attention 1/2/3/4. Utilization drops ~8x, but still insufficient 4. **Activation checkpointing**: recompute activations at backward pass; drops activation usage by a further factor of ~8; still not enough 5. **CPU offloading of activations**: store block inputs off-GPU, prefetch at backpropagation; minimal performance impact; Ryabinin attributes first implementation "to the best of our knowledge" to **Unsloth**; frees ~37 GB 6. **Arctic sequence length training**: tile element-wise computations (losses, MLPs) to avoid 3M-length buffers → 3 million tokens becomes possible ## Primary Contribution: "Untitled Ulysses" - **Key finding**: computing even one set of heads at a time already saturates GPU compute within an iteration - **Method**: schedule multiple heads per GPU in chunks; compute attention per chunk, store partial results, reuse previously allocated buffers across iterations - **Benefit**: smaller reused buffer instead of one huge allocation; saves activation memory **without significant throughput impact at small scales** - **Trade-off stated**: larger chunk → higher memory utilization but faster model run; smaller chunk → the reverse ## Results (as claimed by Ryabinin) - At **8B and 32B scales**, matches the most memory-optimized transformer training implementations closely while scaling further to **5 million tokens**, sometimes more performant at shorter contexts - Stacking techniques plus "U-Pipe" (referenced during results discussion; relation to Untitled Ulysses not fully disambiguated in transcript) frees memory that can be reinvested across pipeline stages, or enables 3M–5M context training ## Takeaways - Bottlenecks "might appear where you least expect" - Recommends **PyTorch profiler** and other tooling - Paper and results are public; upcoming thread planned ## Q&A Clarification An audience member asks whether "QKV" refers to quantization parameters. Ryabinin **corrects**: it refers to the **query, key, and value matrices** of the transformer layer. He explains the challenge: a 3M-length sequence would, in the vanilla approach, require allocating a 3M-sized tensor along one axis — significant enough that no single technique (including U-Pipe) suffices alone. ## Qualifications & Tensions Preserved - Offloading attribution to Unsloth is stated as "**to the best of our knowledge**" — tentative, not settled - The transcript's mention of "U-Pipe" appears alongside "Untitled Ulysses"; the relationship between the two names is not explicitly resolved in the transcript - Throughput claims are qualified as "at small scales" and "sometimes" more performant — not uniform - Flash Attention is referred to loosely as "Flash Attention 1 2 3 4" # Speaker Attribution & Provenance Record **Video:** Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI **Video ID:** `01a15ca9-0e54-737b-b28e-576697e6adfa` ## Speakers | Speaker | Role | Evidence | |---|---|---| | **Max Ryabinin** | **Creator / Presenter** | Self-identified: *"my name is Max. I am VP of research and development at Together AI."* Delivers the full research talk. | | **Unidentified audience member(s)** | **Q&A participants** | Asks questions in the post-talk Q&A (e.g., a question about "QKV quantization parameters," and a remark about joining mid-talk). Names not stated in transcript. | | **Possible moderator/second voice** | **Unconfirmed** | Closing line *"in that case, thank you very much for questions and for listening"* and an earlier *"Do you guys have any questions"* — speaker identity ambiguous; may be Max or a host. | No narrator or guest role present; single-presenter talk with audience Q&A. ## Referenced Sources & Third-Party Claims - **Hugging Face blog post** on model training — cited as source of an example slide showing sequence-length memory impact. - **DeepSpeed Ulysses** (Microsoft) — context parallelism technique, credited as *"first introduced by Microsoft."* - **Unsloth** — credited with first implementing the activation-offload-to-CPU optimization *"to the best of our knowledge."* - **Arctic sequence length training** — cited tiling/chunking technique. - **Flash Attention 1/2/3/4**, **fully sharded data parallelism**, **activation checkpointing** — named existing techniques discussed. - **Together AI paper and upcoming thread** — the presenter's own work ("Untitled Ulysses" / "U-Pipe"), stated as *public* with *more results* and an upcoming explanatory thread. ## Promotional Segments - **Company intro promotion:** Opening section describes Together AI's commercial services — AI-native cloud, GPU clusters, fine-tuning, RL, inference with *"more than 200 models,"* serverless and dedicated deployment. Flagged as **promotional**. - Minor service mentions ("advanced optimizations, which I will not be able to speak about today") — soft promotion. ## Availability & Limits - No curated metadata was supplied; no comment availability information present. - Q&A audience members are unidentified in the transcript. - Full technical results deferred to the paper (not included in this source). Everyone, my name is Max. I am VP of research and development at Together AI. And today, I'm going to tell you about our research project, which is called Road to 5 million sequence length, breaking memory barriers in context parallelism. So, to begin, uh I'll first say a few words about Together AI and who we are. Together AI is an AI native cloud, which provides services and infrastructure for AI developers and builders at all stages of development, starting from uh creating a model, where you just might need a GPU cluster with heavily optimized computes and uh highly reliable uh systems, uh all the way through model shaping, where you can take existing models and customize them for your tasks in terms of performance, in terms of speed, in terms of quality, through services such as the fine-tuning or uh reinforcement learning. Also, we are an inference provider, so if you have an app which is reliant on open-source model inference, uh you can uh work with us and we'll provide you with the fastest way to launch and use AI models with more than 200 models in our portfolio, uh options for deployment, which include serverless and dedicated inference, and a ton of advanced optimizations, which I will not be able to speak about today. Um the purpose of this talk is uh focused on model training, customization, and fine-tuning in particular. And uh I'll start by asking a question. Uh I think in the last few months or like at least a year or so, we're seeing a lot of interest in the community, uh both on the system side and on the research side, in training long context models. Um the primary reasons for that are twofold, I would say. First of all, with the uh like explosion in popularity of agents, you can see a lot of different applications where you might want to put as many tokens as you want in your context, and you want the model to leverage that context effectively. Um second, with the development of applications such as video generation, you might often need to keep track of multiple uh like uh different frames, which or like even uh multiple frames per second, which can um occupy quite a few tokens in your context pretty quickly. And you also need models that have good uh sense of uh temporal consistency, which means that they are able to see what was happening a few seconds or ideally a few minutes ago. Um to do that all effectively, you need to make sure that the models are able to process that context and work with it correctly at the training time. Um but even if you're not at the scales of like millions of uh tokens in the context length, it's still quite important to understand where the memory goes, because who knows, maybe you might be able to reinvest it in some other ways and speed up your training overall. So, the problem here is that if you are taking a standard transformer-based language model and trying to extend its context, you can run into two bottlenecks. Bottleneck number one is that you are faced with quadratic computation, because uh long story short, for transformer-based models, uh you have pairwise interactions across all the elements in a sequence. The second problem is more insidious, one might say. Uh as you continue scaling your context, your memory keeps growing linearly, which is not as bad, but still pretty difficult to deal with, unless you apply a range of specific techniques. Um and this is an example from Hugging Face's blog post on model training, which shows that the sequence length growth can affect your memory limits pretty considerably. Um yeah, here's the slide. And our goal of that project was to see how far exactly we are able to get with a range of existing techniques that are pretty well known to some in the community, as well as some further optimizations that we wanted to leverage to push this a bit further ahead. So, um let's say you're taking a model which is a standard Llama 3B architecture, you're trying to fit 3 million train tokens into your context, and you're taking uh all of this on an 8 uh x H100 GPU node. Um the first stage you'll see is that even with uh just model parameters, you're not able to fit uh it into the GPU. Uh you run out of memory just by trying to place the model. Of course, the next stage is to apply fully sharded data parallelism, where all the parameters are like basically chunked across the eight GPUs that you have, which is great, but uh still doesn't solve the problem. You see that uh the memory usage for the model drops quite significantly, but you still are running out of memory because of all the attention activations. The next point that uh we've leveraged, and I encourage you to use as well, is uh taking advantage of uh context parallelism. In particular, there is a pretty well-known technique called DeepSpeed Ulysses, first uh introduced by Microsoft. The idea is that instead of computing all of your multi-head attention like on every GPU separately for the whole sequence, you can do something more clever. In particular, you can try to compute the attention for different heads at different points in time, uh or like on different GPUs, uh through communicating these activations as uh they are required, in such a way that one GPU is only responsible for one attention head here, but it's still computing the attention over the whole sequence. Um that technique is quite effective at uh addressing the problem, and it also allows you to utilize the best possible attention implementation, like Flash Attention 1 2 3 4, um to optimize that part of the computation. And then you aggregate the results as uh you would have uh previously. So, if you apply Ulysses context parallelism, the utilization drops quite significantly, like approximately 8x uh here, as uh it should, but we are still quite far from our goal of being able to fit that onto just a single uh H100 node. So, what happens next is that we can try to recompute the activations as they uh are needed to us at the backward pass. That technique is known as activation checkpointing, and uh it's available in pretty much all of the deep learning frameworks these days that you could use. You just need to enable it in a correct way that does not uh impose too much of a computational burden on you. Um with that, with activation checkpointing, you can drop the activation usage by like a further factor of eight, but still something else needs to be done. Uh the next optimization is um also connected to the storage of activations. You can try to uh store some of the inputs to each transformer block, not on the GPU, but instead offload them to CPU when they are not required. Uh this is not very impactful for the performance, because you can offload uh it and prefetch when you are trying to backpropagate to that to the corresponding layer. Um this optimization, to the best of our knowledge, was first implemented uh by Unsloth, and uh it allows you to drastically expand the context window. Um the next point is that you're getting uh with offloading next to 37 uh gigabytes of data, but then comes the other part of offload memory usage. Uh what happens next is that you can essentially tile all all of the computations across the sequence length in case they are element-wise. So, all the loss computations, all the MLPs, they can be chunked to avoid creating these huge buffers that would be 3 million along one of the dimensions. Um that's Arctic sequence length training, and even with these optimizations, you are finally getting to a point where 3 million is possible. But, what if you wanted to go further? And here we actually need to do something else, which is the primary optimization we've done in our work dubbed the Untitled Ulysses. And you could describe it as a further deeper analysis and expansion of this context parallelism technique. So, what we found was that even trying to compute one set of heads at a time is already enough to saturate the computational capacity of the GPU with within one iteration. Which means that if you have multiple different heads scheduled to be executed on one GPU, you can divide it in chunks and then essentially iterate through these chunks over time. So, we have one group of heads which are being recomputed, then you compute attention over them, you store the partial result, then you follow up with the next stage, which can reuse all the buffers you've allocated at the previous stage. Um so, the advantage here is that instead of allocating this huge buffer as you would have before like here in this slide, you allocate a buffer which is smaller, but you reuse it across like two or more different iterations. And that allows it to further save on the activation memory for your training without any significant impact to the throughput at small scales. So, here you can see the results that we've measured across different context parallelism techniques. As you can see, both at the 8 billion scale and at the 32 billion scale, we are matching quite closely the most memory optimized implementations of transformer training while being able to scale even further like 5 million tokens and sometimes even being more performant at small at shorter context lengths. Um the relation between the chunk size or the number of heads you compute at the same time and the throughput is quite straightforward. So, if your chunk is larger, your memory utilization is higher, but at the same time you can run the whole model a bit faster. Um so, by stacking all of these techniques together and applying U-Pipe on top, you can, for example, free up a bit of additional memory in your training if you need it and reinvest it somewhere else, for example, among the stages. Or you could say that we're interested in training across not 3 by 5 million context lengths and then you pipe is the technique that will save you by contrast to everything else. So, as a takeaway, I think one of the things that could be quite insightful here is that training models with large context lengths is a very interesting and challenging goal, but the bottlenecks might appear where you least expect. So, tooling like the PyTorch profiler, which we elaborate on on in our paper, or other or other techniques can help you a lot. And also check out our paper for more results. All of that is public at the moment and we have an upcoming thread which will illustrate the method in more depth. Thank you very much for listening and now we are ready for questions. Thank you. Do you guys have any questions cuz we're together and I am employees. So, who did you join it from the middle so we lack some certain context? I'm just curious about the so the QKV that was quantization parameters is that correctly Uh not exactly. It was just the query, key, and value matrices of the transformer layer. So, you multiply them here in the attention part, which creates most of the complexity because all of the queries have to like result in all of these pairwise active interactions with like keys. Uh and the problem is that if you have a a sequence which is like 3 million in length, it means that technically like in the standard most vanilla way, you would have just like allocated that whole big tensor which has 3 million in which is 3 million in size along one of the axis. And like that's pretty significant as you could imagine, which means that you have to resort to like not just one technique, which is U-Pipe, but a range of other approaches to somehow help you like uh leverage like execute these computations without running out of your memory. So, yeah, that's the key idea and the key challenge of working with transformers at the scale. Cool. Um in that case, thank you very much for questions and for listening.