--- schema: 1 type: video title: >- Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI description: >- Training a standard LLaMA 3B model with a 3 million token context on a single 8xH100 node fails before you even start: the model parameters alone exhaust GPU memory. Max Ryabinin from Together AI walks through the full stack of techniques needed to get there: fully sharded data parallelism, DeepSpeed Ulysses context parallelism for an 8x activation reduction, activation checkpointing for another 8x, CPU offloading for transformer block inputs, and chunked sequence training to avoid allocating buffers 3 million tokens wide. Even that stack falls short at 5 million tokens. The novel contribution, Untied Ulysses, goes deeper into the context parallelism step: instead of allocating one large buffer per attention head group, it chunks the heads further and reuses those buffers across iterations, cutting activation memory with negligible throughput impact. At both 8B and 32B scale the results match the most memory optimized transformer training baselines while pushing sequence length 25% further than prior Ulysses implementations. Speaker info: - https://www.linkedin.com/in/max-ryabinin/ - https://x.com/m_ryabinin --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/TUnPNY4E2fw-road-to-5-million-tokens-breaking-barriers-in-long-context-training-max-ryabinin-together-ai Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/road-to-5-million-tokens-breaking-barriers-in-long-576697e6adfa markdown: >- https://liquidgold.cc/channel/aidotengineer/video/road-to-5-million-tokens-breaking-barriers-in-long-576697e6adfa.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T09:46:36.765Z' language: null children: [] videoId: TUnPNY4E2fw durationSeconds: 950 publishedAt: '2026-06-08T17:00:21.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 25 endSeconds: 139 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=25' heldReason: rights - ordinal: 1 text: null startSeconds: 149 endSeconds: 229 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=149' heldReason: rights - ordinal: 2 text: null startSeconds: 229 endSeconds: 239 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=229' heldReason: rights - ordinal: 3 text: null startSeconds: 239 endSeconds: 333 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=239' heldReason: rights - ordinal: 4 text: null startSeconds: 333 endSeconds: 343 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=333' heldReason: rights - ordinal: 5 text: null startSeconds: 343 endSeconds: 446 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=343' heldReason: rights - ordinal: 6 text: null startSeconds: 343 endSeconds: 446 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=343' heldReason: rights - ordinal: 7 text: null startSeconds: 456 endSeconds: 550 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=456' heldReason: rights - ordinal: 8 text: null startSeconds: 456 endSeconds: 550 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=456' heldReason: rights - ordinal: 9 text: null startSeconds: 560 endSeconds: 679 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=560' heldReason: rights - ordinal: 10 text: null startSeconds: 560 endSeconds: 679 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=560' heldReason: rights - ordinal: 11 text: null startSeconds: 691 endSeconds: 796 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=691' heldReason: rights - ordinal: 12 text: null startSeconds: 691 endSeconds: 796 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=691' heldReason: rights - ordinal: 13 text: null startSeconds: 796 endSeconds: 808 watch: 'https://liquidgold.cc/watch/TUnPNY4E2fw?t=796' heldReason: rights --- # Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI Training a standard LLaMA 3B model with a 3 million token context on a single 8xH100 node fails before you even start: the model parameters alone exhaust GPU memory. Max Ryabinin from Together AI walks through the full stack of techniques needed to get there: fully sharded data parallelism, DeepSpeed Ulysses context parallelism for an 8x activation reduction, activation checkpointing for another 8x, CPU offloading for transformer block inputs, and chunked sequence training to avoid allocating buffers 3 million tokens wide. Even that stack falls short at 5 million tokens. The novel contribution, Untied Ulysses, goes deeper into the context parallelism step: instead of allocating one large buffer per attention head group, it chunks the heads further and reuses those buffers across iterations, cutting activation memory with negligible throughput impact. At both 8B and 32B scale the results match the most memory optimized transformer training baselines while pushing sequence length 25% further than prior Ulysses implementations. Speaker info: - https://www.linkedin.com/in/max-ryabinin/ - https://x.com/m_ryabinin --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/TUnPNY4E2fw-road-to-5-million-tokens-breaking-barriers-in-long-context-training-max-ryabinin-together-ai Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/road-to-5-million-tokens-breaking-barriers-in-long-576697e6adfa · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:14 to 2:19) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (2:19 to 3:49) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (3:49 to 5:33) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (5:33 to 7:26) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (7:26 to 9:10) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (9:10 to 11:19) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (11:19 to 13:16) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (13:16 to 15:50) > Held: transcript rights do not permit this chunk's text.