A talk by Olive Song, RL Lead at MiniMax, on how MiniMax M3 was built. The model is described as an open-weight model with agentic coding capability, a 1M token context window, and native multimodality, trained on text and vision from the first step. Topics covered include MSA, MiniMax's sparse attention design (an index branch that selects relevant blocks and a sparse branch attending only to those, reported as ~9x faster prefill and 15x faster decoding than full attention), why existing designs like DeepSeek's DSA did not fit grouped-query attention or GPU memory access patterns, block-level retrieval, efficient kernels, why adding vision later gave unstable training results, attention map analysis, and 3D attention inside the vision transformer. Recorded at AI Engineer World's Fair 2026 in San Francisco. # Channel Identity Observations — MiniMax M3 Technical Talk (Olive Song) ## Explicit Self-Description - **Speaker identifies as a researcher at MiniMax**: *"My name is Olive. I work at Minimax on reinforcement learning research."* - **Team scope claimed**: The presenter describes the team as *"one of the few labs over the world that works on multimodalities,"* citing prior work on **LLMs, speech models, video generation models, and coding agent models**. - **Product claim**: MiniMax M3 is described as *"open weight… with frontier performance"* and *"the first open weight model to combine"* three capabilities: coding/agentic ability, 1M context, and native multimodality. ## Intended Audience - **Technical ML audience**: The talk assumes familiarity with **GQA/MQA, attention mechanisms, top-k selection, GPU memory access patterns, learning rates, CPT-style training**, and references DeepSeek's DSA without background explanation. - **Practitioners evaluating models**: Emphasis on deployment-relevant facts — *"it can drop in for 90% of the coding works that you are currently doing"* and inference speedups (*"nine times faster pre-fill and 15 times faster decode"*). ## Claimed Scope of Content - **Model architecture and training methodology**: Sparse attention design (MSA), algorithm–infrastructure co-design, and native multimodal training strategies. - **First-party experimental evidence**: Attention map analyses, learning-rate sensitivity findings, and 3D-attention-in-ViT experiments with a stated 4-frame trade-off. ## Observable Presentation Patterns - **Promotional framing alongside technical depth**: Claims like *"first open weight model"* and *"frontier performance"* are interleaved with concrete design rationale. - **Educational talk format**: Opens with introductions and a roadmap ("first I'll briefly explain…"), closes with a summary — consistent with a conference/meetup presentation style. ## Notable Distinctions - **Claims vs. demonstrated**: Speedup figures, benchmark claims, and "first open weight" status are **creator claims**; the demonstrated content is the design reasoning (GQA diversity, block-level retrieval, memory contiguity) and training-strategy trade-offs. - No private motivations were stated; scope evidence is limited to this single talk. # Delivery Observations: "How MiniMax M3 Was Built" — Olive Song ## Overall Tone and Register The speaker delivers a **technical talk in a formal presentation register**, structured around three announced "pillars" (coding/agents, 1M context, native multimodality). The tone is measured and explanatory rather than promotional, though capability claims are stated with confidence: > "It can drop in for 90% of the coding works that you are currently doing." ## Verbal Fillers and Hesitations A prominent, consistent feature is **frequent filler words**, especially "um" and "uh," distributed throughout the talk — from the opening to the closing: > "Um all right. Um so, hi everyone. I My name is Olive." > "So, it is nine times faster pre-fill and 15 times faster decode versus full attention. And the third one is that the the the model was trained native multi-model from step zero." The fillers appear at clause boundaries and mid-sentence, suggesting a live, unscripted delivery rather than a rehearsed read-through. ## Word Repetitions and Self-Corrections The speaker **repeats or stumbles on words** when introducing technical terms, which recurs across the talk: > "what we call soda uh coding and agent capabilities" *(apparent restart before "coding")* > "Instead, we took an algorithm algorithm and infrastructure co-design approach." > "From the algo- From the algorithm side, we remove the aggregation..." > "It is is supposed to provide" / "block block level retrieval" / "hardware friendly friendly" This pattern clusters around **new technical concepts**, suggesting the terminology is effortful to produce live even when the underlying ideas are fluent. ## Teaching Habits ### Numbered, pillar-based structure The speaker frames content as a small set of named pillars and enumerates them explicitly: > "it is actually the first open weight model to combine the following three um frontier capabilities. The first one is... And the second pillar is... And the third one is..." ### Question-and-answer framing The speaker poses the problem as an explicit question before answering it — a recurring pedagogical move: > "So, what are the issues if we directly apply the existing sparse attention architecture?" > "So, how did we address these issues in MSA?" > "What does that mean? Our model was trained with both text and vision understanding from step zero..." ### Contrast pairs Content is repeatedly taught via **contrast between two approaches**: > "for CPT-style multimodal models, the text tokens often pay very little attention to visual tokens... Well, in contrast with native multimodal training, the attention map looks much more integrated." > "Compared with token level selection, block level retrieval significantly reduces the number of candidates..." ### Explaining by simplification with concrete numbers Abstract points are grounded with specific figures: > "we can think of the KV 8 layout as one KV head with a large head dimension, for example, 1 by 512... For example, it's 4 by 128." > "using four frames gave us the best best trade-off between performance and efficiency." ## Recurring Vocabulary and Phrases - **"from step zero" / "from scratch"** — repeated to emphasize the native-multimodality message: "trained it natively multimodal from step zero," "trained from scratch with multi-modal data," "from the very beginning." - **"infrastructure" vs. "algorithm"** — used as a recurring dichotomy: "an algorithm and infrastructure co-design approach," "from the infrastructure and implementation perspective." - **Design-principle framing**: "Our core design principle is that a simpler architecture can always lead to better scaling properties." ## Signposting and Pacing The speaker uses explicit transitions and scope-management phrases: > "So, first I'll briefly explain what sparse attention does." > "I I I won't go into the kernel details here today, but this is the system level component..." > "And of course, uh we did many experiments on multimodal architecture choices." The talk closes with a **summary-recap habit**, listing the threads covered: > "So, overall, with our architecture design, with our training strategies that we explored throughout the stages, and with our analysis and ViT explorations, we were able to train the model from scratch..." ## Uncertainty Notes - The transcript lacks audience interaction, so **no conclusions can be drawn about responsiveness to questions or Q&A style**. - Timestamps in the transcript appear to be word-position markers rather than reliable times; **no time-based delivery observations (pauses, pacing speed) are made** here. - Whether the stumbles reflect difficulty or simply live-presentation style cannot be determined from text alone — only their observable frequency is reported. # Position Record: MiniMax M3 Construction Talk **Source:** Video `01a15ca6-adf5-7dcb-b6bd-050ddc189690` **Title:** *How MiniMax M3 Was Built: Sparse Attention and Native Multimodality* **Speaker:** Olive Song (self-identified in transcript as "Olive," working at MiniMax on reinforcement learning research) **Record type:** Expressed positions only. No claims are verified against external data, and no speaker's private beliefs are inferred. --- ## 1. Speaker Framing ### 1.1 Team and Model Context - **Claim:** MiniMax is "one of the few labs over the world" working on multimodality, with published LLMs, speech models, video generation models, and coding agent models. - **Claim:** MiniMax M3 is **open weight** and has "frontier performance." - **Claim:** M3 is "the first open weight model to combine the following three frontier capabilities." ### 1.2 The Three Claimed Pillars | Pillar | Stated Claim | Stated Support | |---|---|---| | **Agentic coding** | Performs automatic task decomposition, works in coding agents, and "can drop in for 90% of the coding works that you are currently doing." | None given in talk | | **1M context** | Handles full code bases; enables "long user interactions." | Trained with MSA; "nine times faster pre-fill and 15 times faster decode versus full attention" | | **Native multimodality** | Trained with vision and language "from step zero," enabling applications such as computer use agents. | Training-strategy comparisons (Section 4) | > **Observation:** The 90% figure and the speedup figures are stated without methodology, benchmarks, or conditions in the talk. They are recorded as the speaker's claims only. --- ## 2. Core Design Principle - **Stated principle:** "A simpler architecture can always lead to better scaling properties." The speaker says this principle was followed "throughout the entire design." - **Stated goal:** "always pursue a cleaner and simpler architecture." - **Qualification:** The speaker frames the work as **algorithm and infrastructure co-design**, not "purely algorithm driven." > **Tension noted:** The principle is stated as "always," which is absolute. The talk also describes added components, such as an index branch with a KL loss and 3D attention in the ViT. The speaker does not reconcile these with the "simpler" framing in the talk. --- ## 3. Minimax Sparse Attention (MSA) ### 3.1 Architecture as Described **Two branches:** - **Index branch** - Performs "lightweight full attention computation." - Uses **4 query heads and 1 key head**. - Trained with a **KL loss** to match the attention scores of the main branch. - Identifies tokens with the highest attention scores. - **Sparse (main) branch** - Attends only to a subset of tokens selected by the index branch. - Uses the selected blocks to determine which keys and values are needed. ### 3.2 Motivation: Limits of Directly Borrowing DSA > **Claim:** DSA was "a very strong design" but is "highly optimized for Deep Seek's own architecture," particularly the combination of MQA and a large head dimension. The speaker states that moving to the more common **GQA** architecture makes direct application "less straightforward," for two stated reasons: **Algorithm-side concern** - With GQA, reusing the same token selection causes all KV heads to "attend to exactly the same selected tokens," which the speaker says "cancels out the diversity that GQA is supposed to provide." **Infrastructure-side concern** - **MQA:** KV layout reads as one contiguous head (e.g., 1 × 512), which is efficient. - **GQA:** The same total dimension split across heads (e.g., 4 × 128) requires "multiple independent reads," which the speaker says "increases memory access overhead and reduces the compute-to-memory access ratio." - **Top-k:** Token-level top-k has "inherent data dependencies," making it "difficult to fully parallel on GPU," and introducing "non-trivial computation and merging overhead." ### 3.3 Solutions Stated 1. **Per-KV-head selection (algorithm):** The index branch outputs are no longer merged into one selected set. Different KV heads "can attend to different sets of tokens." 2. **Block-level retrieval (infrastructure):** Moves from token-level to block-level selection, which the speaker says: - Reduces candidates and lowers top-k and merging overhead. - Is more hardware-friendly, improving memory access efficiency and the compute-to-memory ratio. - Does "not introduce a significant performance drop" (**stated as "we also show,"** but no data is presented in the talk). 3. **Custom kernels:** Designed for both training and inference. The speaker explicitly declines to discuss kernel details: *"I won't go into the kernel details here today."* ### 3.4 Result as Stated - The M3 model was extended to **1M context** training "very efficiently" using this design. --- ## 4. Native Multimodal Training ### 4.1 Problem Framing - **Claim:** A standalone VL model "has relatively limited application scenarios." - **Stated challenge:** How to introduce multimodal capability "without hurting the core text performance." ### 4.2 Strategies Compared **Strategy A: Post-decay multimodal training (CPT-style)** - Most common approach, per the speaker. - **Stated outcome:** Model is "already largely converged," and multimodal data "tends to interfere with the learned text capability and causes degradation." **Strategy B: Multimodal training before the decay stage** - Introduced after the model has seen some text tokens. - **Stated outcome:** "Highly sensitive to the data mixture and training hyperparameters, such as learning rate." - **Qualified finding:** A smaller learning rate reduced the drop in one setting, but "this conclusion does not necessarily generalize to another run." - **Stated constraint:** During large-scale pretraining, parameters "cannot be arbitrarily changed just for multimodal training." **Strategy C: Native multimodal training from scratch (chosen)** - Multimodal capability introduced "from the very beginning." - **Stated outcome:** Under the same text token budget, the approach "almost does not hurt text capability." > *Qualification retained:* The word "almost" is the speaker's. ### 4.3 Attention-Map Observation - **CPT-style models:** Text tokens "often pay very little attention to visual tokens"; visual regions are "mostly dark" in the attention map. The speaker interprets this as multimodality "not fully integrated." - **Native models:** The attention map "looks much more integrated," with no "clear separation between text tokens and visual tokens." - **Interpretation (speaker's):** The model "learns to fuse visual and textual information more naturally from the beginning." > **Note:** This is a qualitative reading of an attention map shown in the talk, not a quantified metric in the transcript. --- ## 5. Multimodal Architecture ### 5.1 Pipeline (as described) 1. Image or video is processed by a **ViT**, producing visual tokens. 2. Visual tokens replace the original image placeholders in the text sequence. 3. Visual and text embeddings are fed together into the language model. The speaker describes this as "relatively standard." ### 5.2 3D Attention in the ViT - **Standard ViT:** Attention is applied within each image or frame. Patches across frames do not interact directly during ViT processing. - **3D attention:** Patches across frames "can attend to each other directly." - **Stated benefit:** Stronger "temporal and cross-frame visual modeling." - **Stated transfer:** The benefit "also transfers to some image benchmarks," which the speaker reads as strengthening "the overall visual representation rather than only improving video-specific tasks." ### 5.3 Frame Limit - **Claim:** 3D attention cannot be extended to an unlimited number of frames because it "would introduce additional system and load balancing challenges." - **Stated finding:** In the speaker's exploration, **four frames** gave "the best trade-off between performance and efficiency." > *Scope note:* This is described as the result of the team's exploration. No additional frame counts or comparisons are given in the talk. --- ## 6. Summary of Speaker's Overall Thesis - **Stated position:** Training strategy is "not just about adding vision capability." It is about integrating multimodality "while preserving its strongest text capabilities." - **Stated position:** Efficiency at 1M context depends on co-designing sparse attention with GPU memory behavior, not on borrowing an existing algorithm. --- ## 7. Unresolved Points and Caveats | Item | Status in Talk | |---|---| | "Simpler architecture always" principle | Stated absolutely; additional components described without reconciliation | | 90% coding task replacement | Claim only; no method or data | | 9× prefill / 15× decode speedups | Claim only; baseline and conditions not detailed | | "No significant performance drop" from block-level retrieval | Stated as shown; no data in talk | | Learning-rate effect on pre-decay multimodal training | Explicitly qualified as not generalizing across runs | | Text capability preservation in native multimodal training | "Almost does not hurt"; no numbers given | | Attention-map integration | Qualitative visual reading | | Four-frame limit for 3D attention | Speaker's exploration result; no comparison data shown | | Kernel implementation | Explicitly deferred by speaker | # Speaker and Provenance Observation: MiniMax M3 Talk ## Video Identification - **Video ID:** `01a15ca6-adf5-7dcb-b6bd-050ddc189690` - **Title:** *How MiniMax M3 Was Built: Sparse Attention and Native Multimodality. Olive Song* - **Format:** Single-speaker presentation with no visible Q&A or panel segment in the transcript. - **Timestamps:** None present in the transcript or supplied metadata. ## Speakers and Roles Only one speaker is identified in the transcript. | Speaker | Stated Role | Affiliation (as stated) | Identity Source | |---|---|---|---| | **Olive** (full name "Olive Song" appears only in the video title) | Presenter, self-described as working on reinforcement learning research | MiniMax | Self-introduction in transcript; surname from title only | - **Presenter role:** The speaker presents the work as a member of the MiniMax team ("our team," "our M3 model"). - **Other voices:** None identified. The transcript ends with a bracketed `[music]` marker, which is not attributed to a speaker. - **Identity uncertainty:** The transcript itself gives only the first name. The surname comes from the title and is not independently verified in the supplied material. ## Promotional Content > **Flag: the presentation is substantially promotional.** It is delivered by the model's developer and promotes its own release. Promotional segments include: - **Product release announcement:** "we just released our Minimax M3, which is open weight with frontier performance." - **Self-described superlative:** "the first open weight model to combine the following three… frontier capabilities." - **Unverified performance claims:** - "nine times faster pre-fill and 15 times faster decode versus full attention" - "It can drop in for 90% of the coding works that you are currently doing" - **Capability claims:** 1 million context window; "handles full code bases"; automatic task decomposition. - **Comparative claim against other models:** Native multimodal training is described as "very different approach from current open weight models." - **Company self-promotion:** Statements about the team as "one of the few labs over the world that works on multimodalities," along with references to the company's prior LLM, speech, video generation, and coding agent models. These are **speaker-originated claims**. This record does not verify, benchmark, or evaluate them. ## Referenced Sources and Third-Party Claims ### Named Systems and Architectures - **DSA** (described as "DeepSeek's own architecture" context): The speaker says DSA "was a very strong design" but "highly optimized for Deep Seek's own architecture," especially its MQA and large head dimension combination. This is the speaker's characterization. - **GQA** and **MQA:** Architectural references used for comparison. - **MSA (Minimax Sparse Attention):** The company's own method, presented as the speaker's team's design. - **ViT:** Vision encoder referenced as the image/video processing component. - **CPT-style multimodal training:** A training approach the speaker attributes to common practice ("the first and most common approach"). ### Internal Observations Presented as Evidence - **Attention map analysis:** The speaker describes a visualization comparing CPT-style and native multimodal models. No figure is reproduced in the transcript, so the visualization cannot be checked from the transcript alone. - **Unreleased experimental results:** The speaker states that training variations were tested "at scale," that results were "highly sensitive" to hyperparameters, and that 3D attention with four frames gave "the best trade-off." No experimental data or citations are provided. - **Kernel details:** Explicitly withheld ("I won't go into the kernel details here today"). ### Comparative Claims About Other Models - Current open-weight models are described as taking a different multimodal approach. No specific models are named. - The speaker states that a "standalone VL model has relatively limited application scenarios." This is an opinion. ## Provenance Limits - **Cited sources:** No papers, links, or bibliographies are mentioned in the transcript. - **Comment availability:** No information supplied. - **Curated metadata:** Only the title is supplied. No description, speaker bio, chapter markers, or verified channel information is present. - **Transcript quality:** The transcript contains repeated words, self-corrections, and escaped characters (`\>`). Some passages are unclear, so minor wording may differ from the spoken audio. ## Self-Check - [x] Speaker identity and role are explicit in the transcript (first name) and title (surname). - [x] Promotional content is flagged. - [x] Referenced systems and speaker-originated claims are recorded without evaluation. - [x] Missing provenance channels (citations, comments, timestamps) are noted. - [x] No quality judgment on the model, benchmarks, or methods is included. \>\> Um all right. Um so, hi everyone. I My name is Olive. I work at Minimax on reinforcement learning research. And today, I'm going to talk a little bit about um our new model, Minimax M3, and how we trained it natively multimodal from step zero, and how we trained it with MSA, which is called uh which is Minimax sparse attention. Um a little bit introduction of our team. Uh we are one of the few labs over the world that uh works on multimodalities, which means that we have published our um of course LLM models before. Um we worked on our speech models. We have our own video generation models, and we have our coding agent models. And lately, we just released our Minimax M3, which is open weight uh with frontier performance. So, it is actually the first open weight model to combine the following three um frontier capabilities. The first one is what we call soda uh coding and agent capabilities. The model is able to do a lot of the uh agentic tasks that you imagine. Um it it can do automatic task decomposition. It can work in your coding agents. It can drop in for 90% of the coding works that you are currently doing. And the second pillar is that the model has 1 million context window. Um so, which is very long that enable like unlocks pretty much base all basic user long user interactions um of like it handles full code bases. And we trained it with MSA Minimax sparse attention. So, it is nine times faster pre-fill and 15 times faster decode versus full attention. And the third one is that the the model was trained native multi-model from step zero. Which means that from step from scratch, we combined visual understanding with um language generation. So, the model would understand both capabilities very well from the start. And uh so that we can build many more applications on that, say computer use agents, right? So, first I'll briefly explain what sparse attention does. Our core design principle is that a simpler architecture can always lead to better scaling properties. This is the principle we have followed throughout the entire design. So, our goal is to always pursue a cleaner and simpler architecture. The sparse attention consists of two branches. The first one is what we call the index branch. Which performs a lightweight full attention computation. It uses four query heads and one key head. And it is trained with a KL loss to match the attention scores of the main branch. The index branch is used to identify the tokens with the highest attention scores. These selected tokens are then attended by by the main branch. In other words, the sparse branch does not attend to all of the tokens. But only to a subset of them. This subset is selected through the lightweight computation of the index branch. Then, the sparse branch would use the blocks selected by the index branch to to determine which keys and values are needed for attention computation. All right. And what I want to emphasize today is that our design is not purely algorithm like algorithm driven. Instead, we took an algorithm algorithm and infrastructure co-design approach. Um a good good example to start with is DSA. DSA was a very strong design, but it is also highly optimized for Deep Seek's own architecture, especially the combination of MQA and the large head beam. However, when we move to a more commonly used GQA architecture, directly applying the same design becomes less straightforward. It introduces several practical issues, especially from the infrastructure and implementation perspective. So, this motivates our design choice. Instead of simply borrowing an existing sparse attention design, we re-think the architecture under the constraints of GQA and real system efficiency. So, what are the issues if we directly apply the existing sparse attention architecture? First, from the algorithm perspective, GQA has multiple KV heads, which should ideally provide more diversity across different KV groups. However, if we directly reuse the same token selection strategy, multiple KV heads will end up attending to exactly the same selected tokens. This essentially cancels out the diversity that GQA is is supposed to provide. Second, from the infrastructure perspective, this design is not very friendly to GPU memory access. GPUs are much more efficient when reading contiguous data. In the MQA setting, we can think of the KV 8 layout as one KV head with a large head dimension, for example, 1 by 512. This is relatively contiguous and efficient to read. However, in GQA, the same total dimension may be split across multiple KV heads. For example, it's 4 by 128. These are no longer one single contiguous read. Instead, the GPU needs to perform multiple independent reads across different KV heads. This increases memory access overhead and reduces the compute-to-memory access ratio. And another issue here is the top-k computation itself. To select the most important tokens, the indexer needs to compute top-k scores over a large number of token-level candidates. However, top-k has inherent data dependencies. So, it is difficult to fully parallel on GPU. As a result, token-level selection introduces non-trivial computation and merging overhead. So, how did we address these issues in MSA? From the algo- From the algorithm side, we remove the aggregation of the index branch outputs. In the original design, the results from all index branch In index branch heads are merged into a single set of selected tokens. But, as we discussed earlier, this would remove the diversity brought by GQA. Instead, we keep the selection different different across KV heads. Under our sparse attention design, when using GQA, different KV heads can attend to different sets of tokens. This helps preserve the diversity of GQA rather than forcing all heads to share the same sparse tension. For the infrastructure side, we move from token level selection to block level retrieval. Compared with token level selection, block block level retrieval significantly reduces the number of candidates and lowers the overhead of top key computation and result merging. At the same time, block level retrieval is more hardware friendly friendly. It improves memory access efficiency and increases the compute to memory access ratio. Importantly, we also show that the design does not introduce a significant performance drop. Finally, we design efficient kernels for both training and inference. I I I won't go into the kernel details here today, but this is the system level component that makes the sparse attention design practical and efficient. So, with this design, we were able to extend the training of our M3 model to 1 million context very efficiently. Um and as we talked about earlier, the third pillar is the native multi-modality. Our model, Minimax M3, was trained from scratch with multi-modal data. What does that mean? Our model was trained with both text and vision understand standing from step zero, which is very different approach from current open weight models. Our starting point is very clear. At the current stage, a standalone VL model has relatively limited application scenarios. But combining it with with, you know, text understanding during the training can be challenging. So, the key question and challenge there was how can we introduce multimodal capability without hurting the core text performance? To answer this, we actually explored several training strategies at scale. The first and most common approach is to add multimodal training after the pre-training decay stage, which is essentially a CPT style approach. However, at that point, the model has already largely converged. Adding multimodal data at this stage tends to interfere with the learned text capability and causes degradation. Another approach is to enter multimodal training before pre-training the decay stage. For example, after the model has already seen a number of amount of text tokens. We also tried this direction, but found that the results were highly sensitive to the data mixture and training hyper parameters, such as learning rate, for example. In one setting, a smaller learning rate might reduce the performance drop, but this conclusion does not necessarily generalize to another run. More importantly, during large-scale pre-training, these parameters cannot be arbitrarily changed just for multimodal training. And this led us to our final strategy, native multimodal training from scratch. In this setting, multimodal capability is introduced from the very beginning rather than being added after the text model has already formed its structure. Under the same text token budget, we find that the this approach almost does not hurt text capability. We actually did many fun analysis during these experiments. Um one of them is this interesting observation uh from the attention maps. You can see from this picture that for CPT-style multimodal models, the text tokens often pay very little attention to visual tokens. The visual token regions are mostly dark in the attention map. This suggests that the multimodalities are not fully integrated. Well, in contrast with native multimodal training, the attention map looks much more integrated. We do not see here a clear separation between text tokens and visual tokens. This suggests that the model learns to fuse visual and textual information more naturally from the beginning. So, overall, our training strategy is not just about adding vision capability. It is about integrating multimodality into the model while preserving its strongest text capabilities. And of course, uh we did many experiments on multimodal architecture choices. Our overall architecture is relatively standard. The image or video is being processed by a ViT, which produces visual tokens. These visual tokens are then inserted into the language model input by re- replacing the original image placeholders in the text sequence. After that, the visual embeddings and text embeddings are fed together into the language model. One important finding is that applying 3D attention inside the ViT is very helpful for vision understanding. In a standard ViT setting, attention is is applied within each individual image or frame. For video inputs, this means that visual patches from different frames do not directly interact at the ViT stage. But with 3D attention, patches across different frames can attend to each other directly. This gives the model stronger temporal and cross-frame visual modeling capability. And we found that this improves visual understanding, and the benefit also transfers to some image benchmarks. Suggesting that it strengthens the overall visual representation rather than only improving video-specific tasks. And one practical detail is that we cannot simply extend 3D attention to an unlimited number of frames because that would introduce additional system and load balancing challenges. In our exploration, using four frames gave us the best best trade-off between performance and efficiency. So, overall, with our architecture design, with our training strategies that we explored throughout the stages, and with our analysis and ViT explorations, we were able to train the model from scratch with multimodality understanding. And that would be the end of the presentation today. Thank you. \>\> \[music\]