Marah Abdin and Robert McHardy of poolside discuss manufacturing synthetic code data and pre-training at scale. The synthetic data pipeline pairs templates with supplementary context, varies phrasing, and tunes task difficulty; multistage pipelines port existing data into new shapes (swapping character styles or plots, turning single prompts into multi-turn chats) while an orchestrator filters failed generations. Pre-training follows a "trust nothing" approach: replica runs must match or the run is killed, exposing failures like spiky loss from broken GPUs, a tensor-parallel numerical precision bug, and silently corrupted gradients from a race condition. The talk covers their 118-billion-parameter agentic coding model, whose early results reportedly edge out GLM 4.5 Air on a recipe that held as it scaled. # Video Identity Observations: poolside Talk — Synthetic Data & Pre-Training at Scale ## Observable Presentation Evidence ### Presenters & Affiliation - Two speakers from the same organization, **poolside** (spoken as "Pulsar" in transcript): - **Marah Abdin** — introduces herself as being from the company's **data team**; covers synthetic data. - **Robert McHardy** — covers **pre-training, architecture work, and distributed training**; states: *"I specifically focus on architecture work on and distributed training and so on."* ### Organization Self-Description (Explicit Claims) - The company has **recently shifted from enterprise-only releases to open releases**: > "we've switched from recently from releasing our models towards enterprise to also releasing towards everybody." - Two **open-weight models** released on Hugging Face: **Laguna M** and **Laguna XS** (also referred to as "Laguna XS 2" / "XGen-2"), plus a public **tech report**: *"which has a ton of detail if you're interested."* - A third model (**Laguna S**, 118B total / 8B active parameters) is teased as forthcoming: *"This model is not public yet, so this is a preview."* - Stated strategic focus: **"we want to build the strongest agentic coding models."** Robert notes the model is *"one of the most competitive models for its size and for coding specifically."* ### Intended Audience - A **technical conference/audience** familiar with ML training concepts (tensor parallel, BF16/FP8, distributed data parallel, evals like BigCodeBench, SWE-bench) — no introductory framing is provided. - Audience is assumed to follow open-weight ecosystem references: Hugging Face, GLM, DeepSeek, Nemotron, Llama ("turn 360"), DeepChem FP8 kernels. ### Claimed Scope of the Talk - **Synthetic data pipeline design** (Marah): rephrasing, cross-domain porting (math→code), multi-stage workflows, a modular six-component pipeline framing, and an internal generation infrastructure called **"Hive."** - **Pre-training at scale** (Robert): model replica **weight-hash checks**, silent GPU corruption, exploding gradients from BF16 accumulation, FP8 race conditions, and benchmark previews of Laguna S. ## Creator Claims vs. Demonstrated Patterns | Type | Evidence | |---|---| | **Claim (capability)** | Laguna XS is *"one of the most competitive models for its size and for coding specifically"* | | **Claim (strategy)** | Synthetic data is *"a way to complement [organic data]"*, not replace it; *"at least at Pulsar we don't see it as a way to replace organic data"* | | **Claim (openness)** | Open weights, free download, public tech report, public PR with a DeepChem kernel fix | | **Demonstrated pattern** | Deep technical transparency: sharing previously unreleased failure modes — *"things that we've not shared before publicly like this"* (silent data corruption, ~0.5% gradient corruption from race condition) | | **Demonstrated pattern** | Openness to community contribution: fix *"is public on that QR code if anyone is interested"*; closes with links to **careers page and Twitter** | | **Honest caveats (claim about own limits)** | *"it's not the strongest model in the world"* — concedes MMLU Pro gap vs. Nematron and DeepSeek, attributed to a *"data gap"* | ## Recruitment / Channel Positioning Signals - Explicit **hiring signal** at the close: *"I added also a link to our careers page and our Twitter page if you want to check it out."* - Positions the channel/organization as **research-transparent and open-weight-oriented**, targeting practitioners interested in training infrastructure, synthetic data, and agentic coding. ## Notes on Source Reliability - Transcript contains speaker self-identifications and model claims as spoken; some model names appear phonetically garbled ("Pulsar" vs. poolside, "Luganem," "XGen-2" vs. "XS 2") — observations above reflect the transcript as supplied without external correction. - No curated metadata was provided; all evidence derives from title and transcript. # Delivery Observations ## Overall Format - **Two-speaker talk** with a clean handoff. Marah covers synthetic data; Robert covers pre-training infrastructure and scale. - Handoff is explicit: > "Okay, that's uh, that's it for me and Zendaya. I hope you learned something. Handing over to Robert." (Note: "Zendaya" appears to be a verbal slip or mis-transcription; the speaker is Marah. Uncertain.) ## Tone - **Conversational and informal** throughout both halves. Filler words ("uh," "um," "you know," "kind of," "like") are extremely frequent: > "Um, and you can see here that uh you know, this is a take this ablation this is an ablation result so take the numbers with a grain of salt" - **Casual, blunt phrasing** from Robert: > "If you've got data that sucks, you can't train a good model. If you've got a training code base that sucks, you also can't." ## Recurring Phrases - **"kind of"** — used constantly by both speakers as a hedge/softener (e.g., "a kind of a token uniqueness problem," "a modular way," "treat different data mixes holistically" region). - **"Um, so"** as a recurring transition restart. - Robert's repeated framing of problems as things that **"happen at scale"**: > "some of the surprising things that happen at scale" / "the first example here of uh that happens at scale" / "this is also something you can only observe at scale" - Repeated **"right?"** as a listener-check by Robert ("They're exactly the same run, right?", "you can verify, right?"). ## Teaching Habits - **Analogies and everyday examples**: Marah uses the novel-writing example to explain multi-stage pipelines ("generate, you know, the setting, the character names... You will absolutely get a better novel"). - **Rule-of-thumb heuristics** stated as portable advice: > "the rule of thumb is if task is too hard for your model, then your model will start to fall on its face. Lose correctness, lose diversity. So break down the task, make it simpler." - **Numbered/structured enumeration**: "there's three things that we did on the data side"; the six synthetic-data pipeline components; the four pipeline "shapes." - **Visual anchoring to slides**: both repeatedly reference "you can see here," "on the left-hand side," "on the right-hand side," "the purple one," "the orange curve." - **Caveats before claims** — Robert consistently hedges results: > "take the numbers with a grain of salt" (Marah) > "these are base model evals, right? They are partly indicative... but also not perfectly, right?" ## Transparency & Uncertainty Signals - Both speakers repeatedly flag **limitations and honesty about failure**: - "we were got unlucky and we had a broken GPU" - "it doesn't actually have anything to do with scale, so it was just unfortunate" - "it's not the strongest model in the world, right?" - Robert frames his team's ethos as skepticism: **"we don't trust anything."** ## Handoff and Closing Patterns - Both speakers close with **practical pointers to resources**: tech report, Hugging Face, QR code to a PR, careers/Twitter links. - Repeated emphasis that models are **"open weights... free"** appears in both halves — a recurring talking point. ## Isolated Moments (insufficient repetition) - Marah's "and Zendaya" — single occurrence, likely a slip; cannot establish a pattern. - Audience-directed humor ("this is how you police them, basically") appears once; tone is generally straightforward rather than comedic. # Position Extraction: The Messy Reality of Scale — Synthetic Data and Pre-Training **Video:** *The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside* ## Speakers - **Marah Abdin** (poolside, data team) — synthetic data portion - **Robert McHardy** (poolside, architecture/distributed training) — pre-training portion --- ## Marah Abdin — Synthetic Data ### Context and Releases - poolside shifted from enterprise-only releases to public releases; two open-weight models (**Laguna M** and **Laguna XS**) are on Hugging Face, with a detailed tech report. - The talk's focus is the transition from **M point one** to **point two**, described as a generation switch with "quite a few things" changed. ### Three Data-Side Interventions at Scale 1. An **auto-mixer** for cheap sweeps on clusters of datasets before expensive experiments. 2. Rethought **web data sampling** for higher recall. 3. Heavier reliance on **synthetic data** in several forms. ### Position on Synthetic Data - Abdin explicitly rejects framing synthetic data as a replacement for organic data: > "At least at poolside we don't see it as a way to replace organic data... it is a way to kind of complement it." - Rationale: organic data contains **implicit rationale, planning, and structure** not presented optimally; synthetic data extracts these features and "projects them on new planes," filling gaps and regularizing token presentation and model teaching. - **Pre-training mix: 13% synthetic** (pre-training only, before post-training) for "point two"; since then, a **six-trillion-token synthetic corpus**, continuously growing. ### Problem: Repetition at Scale - Earlier focus on **quality over quantity** led, once scaling training budgets, to **non-optimal repetition of high-quality data**, saturating the model too early. - Solution: **rephrasing** (a common synthetic form), using a generic multi-model scalable pipeline plus two **specialized pipelines** (markup-to-code-and-text, and one for scan data), pushed further on **STEM documents**. - Ablation results: "take the numbers with a grain of salt," but the **diff between repeated seeds and de-duplicated/rephrased tokens persists consistently**. ### Modular View of Synthetic Pipelines Every pipeline composes from six components: **seeds, primary inputs, metadata, secondary inputs, generator function** (agent with tools or prompt templates), and **supplementary functions** (filters, validators). - Complexity spectrum: cheap, seed-heavy, small-model pipelines (e.g., rephrasing) at one end; orchestrated complex workflows for high-value data (e.g., educational data) at the other. - Key rule of thumb: > "If the task is too hard for your model, then your model will start to fall on its face. Lose correctness, lose diversity. So break down the task, make it simpler." **Four pipeline shapes:** 1. **Rephrasing** (covered above) 2. **Multi-stage pipelines** — e.g., generating a novel by first producing setting, characters, plot, twists, then chapters: "You will absolutely get a better novel." 3. **Cross-domain porting** — e.g., translating code; converting math problems to code. 4. **Multi-turn roles** — e.g., two agents in multi-turn chat; judge/evolver iteration loops. ### Infrastructure: Hive - A configurable generation system: a **queue of agents** (each with prompt, parameters, model, inputs/outputs, entry/exit/frequency configuration), **orchestrators** between agents (policing, dynamic instruction changes, agent selection/skipping), and a **supervisor** that polices the orchestrator with a global view. --- ## Robert McHardy — Pre-Training at Scale ### Framing - Echoing a prior speaker's point on treating data mixes holistically, McHardy argues **data, training code correctness, and implementation must be treated holistically**: > "If you've got data that sucks, you can't train a good model. If you've got a training code base that sucks, you also can't." - Team stance: **"We don't trust anything"** when scaling to hundreds of billions of parameters on thousands of GPUs. ### Model Replica Hash Checks - Multiple model replicas (distributed data parallel) should have **identical weights** — verified via periodic weight hashes; mismatch crashes training. ### Incident 1: Broken GPU - Two runs, **identical config, data, and implementation**: one had a broken GPU causing **silent data corruption**, producing spiky loss curves and huge gradient norms. Caught by the hash checks. ### Incident 2: Exploding Gradients (BF16 accumulation) - Initial Laguna M 1 run (~50,000 steps) stopped converging. Cause: activations grew before the **LM head / unembedding**; the tensor-parallel accumulation was performed in **BF16**, which lost numerical precision as activation scale grew — blocking learning and back-propagating into the whole trunk. - Fix: moved accumulation to **FP32**; from that checkpoint, the model resumed converging and gradient norm decreased. ### XGen-2 (XS point two) - Open weights, free; described as "one of the most competitive models for its size and for coding," focused on **agentic coding**. - Recipe combined: increased diversity, reduced repetitions, numerical fixes, more observability, and training/architecture optimization. ### Laguna S (Preview — not yet public) - **118B total / 8B active parameters, 30 trillion tokens, 4,000 GPUs** — a scale test bed for whether data/architecture improvements and numerics hold. - **Incident 3: FP8 race condition** — with DeepChem FP8 kernels (open source), observed illegal memory accesses and NaNs in gradients, traced to those kernels. Unobservable effect: **~0.5% of gradients silently corrupted** (replaced by random values). Fix exists in a public PR, **not yet merged** into DeepChem. - Noted blind spot: hash checks can't catch this, since real runs have no redundant replica with same weights/data to compare forward/backward behavior; a **dry-run hash checker** is in development. ### Early Eval Results (with caveats) - Caveat stated explicitly: these are **base model evals**, "partly indicative... but also not perfectly" — post-training remains, results won't translate one-to-one. - Coding benchmarks (**MultiPL-E, BigCodeBench**): Laguna S beats XS 2 and the larger M.1, and outperforms **GLM 4.5 Air, Qwen 3 60 ("then turn 360" as transcribed), DeepSeek V4 Flash Max** (both noted as recent; the latter "a fair bit larger"). - **BigBench Hard / EvalPlus**: competitive/close, not top. - **SWE-bench agentless multilingual** (their proxy for agentic performance during pre-training): performs **much better than all other tested models**. - Concession: **not the strongest model overall** — on **MMLU Pro**, Nemotron and DeepSeek "perform much better," which McHardy attributes to a **data gap** they "could plug if we wanted to," deprioritized because the goal is strongest **agentic coding** models. - Conclusion: "the recipe held, it scaled, and we will continue scaling it from here"; Laguna S will be released **open-weights, free, "relatively soon."** --- ## Preserved Qualifications and Tensions - Synthetic data positioned as **complement, not replacement** — a bounded claim, not a prediction it will never replace organic data. - Ablation numbers explicitly flagged as uncertain ("grain of salt"). - Base-model evals flagged as **imperfect predictors** of final post-trained models. - Model comparisons include admissions of weaker performance (MMLU Pro) framed as a deliberate prioritization, not pure capability. # Video Attribution Record — poolside Talk ## Video Identification - **Title:** The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside - **Video ID:** `01a15ca8-39e3-7e0b-b0af-f70cf967d8f6` - **Format:** Conference/talk presentation with two speakers ## Speakers & Roles | Speaker | Role | Segment | |---|---|---| | **Marah Abdin** (transcribed "Mara Abden") | **Co-presenter / creator-side speaker**, poolside data team | First half — synthetic data | | **Robert McHardy** | **Co-presenter / creator-side speaker**, pre-training, architecture, distributed training | Second half — pre-training | | Unknown attendee | Closing "Hey." | Final line after music; role unidentified | - No guests, narrator, or third-party speakers identified. - Marah references "the previous talk" speaker's point about holistic data mixes — **external, unnamed speaker**, claim referenced not quoted. ## Referenced Sources & Third-Party Claims - **poolside models/artifacts** (self-referenced): Laguna M, Laguna XS, "XGen-2" / XS point two (transcript uses inconsistent naming), tech report on Hugging Face, upcoming ~118B-parameter model (non-public preview). - **DeepSeek FP8 kernels** — open source; race condition claim with silent ~0.5% gradient corruption; fix available via QR code link, in PR, not yet merged. - **Comparison models named by Robert:** GLM 4.5 Air, "GLM turn 360" (transcription uncertain), DeepSeek V4 Flash Max, Nemotron. - **Benchmarks cited:** Multiple Evals code bench, Big Code Bench, Big Bench Hard, Eval Plus, SWE-bench agentless multilingual, MMLU Pro. - **"Beyond Web"** — referenced by Marah as an example of rephrasing pipelines (transcription uncertain). - Robert explicitly **caveats** ablation numbers ("take the numbers with a grain of salt") and base model evals as partly indicative only. ## Promotional Segments (flagged) - Opening introduction of poolside models, open-weight releases on Hugging Face, and tech report. - Closing: **links to poolside careers page and Twitter page**. - Emphasis that models are "open weights… download and use it for free" (repeated). ## Provenance & Availability Notes - Some learnings described as **not previously shared publicly** (model replica hash checks, broken GPU incident, exploding gradients, FP8 race condition). - Upcoming Laguna S-class model: **not yet public** at time of talk; preview only. - **No curated metadata supplied**; no comment availability information present. - Transcript contains transcription artifacts ("Pulsar" for poolside, "Zendaya," inconsistent model names) — uncertainty noted, not corrected beyond flagging. Hi everybody. Thanks for coming to our talk. Uh my name is Mara Abden. I'm I'm from Pulsar. I don't know if you're familiar with our models. Uh our data team and today my colleague Robert and I will be talking a little bit about um some of the challenges that we've seen as we scale our models over here. Uh particularly if you haven't heard, we've switched from recently from releasing our models towards enterprise to also releasing towards everybody. We've actually put out two open weight models. They're on Hugging Face a few weeks ago. We have Laguna M and Laguna XS. We also put out a tech report which has a ton of detail if you're interested. Um as you can see here by the Laguna M point one and XS point two, this uh this is this is actually is because we have switched out quite a few things between those two models and so big flavor of this talk is going to be about kind of how did we transition from one to two. And in fact, we've continued to do so and now we actually have a newer version and Robert will give a sneak peek about soon to be released model. Okay. So I will be particular talk about synthetic data part of things. So um there's three things that we did on the data side to kind of resolve some of the issues that we've seen with scale. One is that we implemented auto mixer that basically just gives us a chance to do a cheaper sweep on clusters of our sets before moving on to more expensive experiments. And then we improved we just rethought our sampling of web data for higher recall. Um and then the third one is that we relied a lot more on synthetic data in a few forms which I'm going to. Okay, so before we kind of going into what does that mean and what have we done, etc. One uh why would we kind of It's It's Sometimes it's fair at least to ask why synthetic data and the thing is that uh at least at Pulsar we don't see it as a way to replace organic data. I don't see it so in the current state of the world at least, but uh it is a way to kind of complement it. And the thing is that organic data has a lot in it that is basically kind of implicitly hidden. A lot of things that could teach the model or not very presented in the most optimal way sometimes. And so synthetic data gives us a track to extract some of these features and project them on some new planes. And this is how we get to expose um implicit rationale, implicit planning, implicit structure, and a way for us to fill gaps and regularize not only how we present the tokens, but also how we are teaching the model. Um for access point two in particular, we settled on 13% of the mix. This is only pre-training stages before post-training. Um and since then we've just been continuously generating more data in a bunch of directions. Now we have a six trillion token uh corpus that's continuously growing. Okay. Um so yeah, so uh what kind of what I I I I just said is that we saw some some I guess limitations switching from Luganem to uh point one to our point two models. And so one of those things is that we basically we started on data and this is really not a crazy kind of problem. We very intuitively started from a place on a smaller scale where we were um we were basically focusing on quality versus quantity um maybe a little too much because eventually when we started scaling our models, we had to scale our training budget and with that uh came some limitations because we started hitting repetition uh like non-optimal repetition on some of our high-quality data which saturated the model a little too early. So um we One of the ways we've particularly for this like a token uniqueness problem uh, we relied on which is a very common form of synthetic data rephrasing which you know you just may heard in like Beyond Web for example. It's become pretty trendy these days. Um, and you can see here that uh you know, this is a take this ablation this is an ablation result so take the numbers with a grain of salt but what persists pretty consistently is the diff between using the orange you would be just the seeds with repetition and then the green would be replacing some of those repeated tokens with higher like with the multi-mode reverse or at least yeah, all of them or at least reducing the repetition. Um, and so uh for rephrasing particular we did to the you know, what everyone's doing with the you know, generic kind of multi-mode very scalable pipeline but we also took it a step far and we did two other specialized pipelines one from to go from Morocco to code and text and one to go specifically for scan data. Just because this is a very cheap scalable pipeline so it kind of you kind of have to rely very heavily on the seed and we push a little further on that for the stem documents. Okay. So okay, so if you kind of think of everything is a kind of an modular way you can think of every synthetic data pipeline is composed of the same six components and so you have your seeds, your primary inputs, your metadata, your secondary inputs, your generator and function which can be an agent with tools or one you know, with some prompt with some prompt templates and then some supplementary functions like filters and validators and so on. And really you can compose just about all pipelines from like very simple to very to very expensive pipelines like this. And on on that note and kind of we've we have covered quite a bit of wide scope on the the axis complexity and you kind of can think of it if one end you have like the cheap scalable pipelines that have used smaller models and can get get with it because they're seed heavy examples of phrasing. And then on the other end you have more complex pipelines with a little more orchestration in the workflows. This is reserved when the we're building on something that's worth it. Educational data. But really it's this is how we're not blocked or limited by whatever teacher model can do. And this is how we can be ambitious in our synthetic data. Because the rule of thumb is if task is too hard for your model, then your model will start to fall on its face. Lose correctness, lose diversity. So break down the task, make it simpler. And yeah, I will give some examples of kind of shapes rather than just like something more concrete about how do we use this this modularity. And one shape is the formula writing is just rephrasing. We already talked about this. Multi-stage pipelines and multi-stage workflows. Basically this is what I also just said. You take a step and you break it down into multiple steps. You can aggregate the processing. Slowly build up the the generation. Example of this were if you wanted to generate a novel for example, it's you could generate one chapter at a time, but you could also, you know, take it a little slowly. One first like generate, you know, the setting, the character names, the character styles, the plot, you know, some twists. And then and then from there go into generate the chapters one by one. You will absolutely get a better novel. Um Okay, third is cross-domain porting which really is just like like moving from one mode to another. Example would be changing like translating code. Another example would be something we did which is take our math problems and convert them to code. The last one is multi-turn role. What I mean by that all I mean is that instead of having kind of a very one like very singular or or linear or even non-linear kind of view of things you have more of an iteration. This encapsulates pretty much everything. And and that like an example of that would be multi-turn chats when you have two agents talking to each other or a task evolution pipeline where, you know, you have a judge and an evolver going back and forth for some k amount of time. Um, and so on. Okay. So, lastly, uh, forewarning off to Robert, I do want to kind of just mention that we because of the modularity of the way we think about this, we can implement an infrastructure that's pretty configurable. Um, so this is how we present Hive. Hive is basically a way for us to easily, um, that uh, to easily construct uh, generations where now you have uh, a queue of agents that you define. Each one has, you know, its prompt, its parameters, its model, etc., inputs, outputs. But it also has when you can configure when enters the queue, when it exits, how many have frequency to come in. And then we have orchestration in the middle between agents. And with this orchestrators are really useful because they give you more flexibility and kind of presenting a hierarchy between LLMs that are generating is very uh, it's this is how you police them, basically. But also give them some some form of creativity and dynamically change instructions for the next agent or choose which agent goes next, which agent skipped, and so on. Um, and lastly, you have the supervisor, which basically someone who polices the orchestrator and has more of a global view. Um, cool. Okay, that's uh, that's it for me and Zendaya. I hope you learned something interesting. Handing over to Robert for uh, pre-training stuff. \>\> All right, thank you, Mara. Um, cuz I will talk a little bit more on the um, actual pre-training side rather than just data. I liked in the previous talk, um, the the speaker made a point that we should treat different data mixes holistically, different training stages. I want to make the same point that we should treat uh, data and implementation of your training code base, uh, correctness of it, and so on, also holistically. If you've got data that sucks, you can't train a good model. If you've got a training code base that sucks, you also can't. Um, so I specifically focus on architecture work on and distributed training and so on. Um and the way we we look at things in my team is uh we don't trust anything. There's so many things that can go wrong when you scale models to billions of parameters to hundreds of billions of parameters um training on thousands of GPUs and so on. And I want to show you some of the learnings that we got from training Laguna m.1. Um And yeah, some of the surprising things that happen at scale. Um so one thing we do is we've got these um model replica hashtags. So essentially when we train a model we've got multiple replicas of the same model, right? Um distributed data parallel. And uh we know there's an invariant, the weight should always be the same across all of these replicas. That's something you can verify, right? We you can calculate a hash over the weights and you know that should always be the same across all replicas. So we do that in training and periodically compare them. Um if all of these hashes are identical, then we know we can continue training. If they're not identical, we know something has gone seriously wrong uh because that should never happen. And we crash the training. And I'll give you some examples now of things that we've not shared before publicly like this. Um so I hope they're interesting. Um so the first example here of uh that happens at scale are broken GPUs. On the left-hand side we've got two loss curves uh and on the right-hand side the corresponding uh gradient norms um that we observed during training. And you can see that these loss curves look quite different, right? Like the purple one has got quite some bumps, looks a bit spiky. The gradient norms are huge for that run. And there's actually no difference uh in terms of model configuration, training data, training implementation um between these runs. They're exactly the same run. Just in one of them we were got unlucky and we had a broken GPU included. That broken GPU caused silent data corruption and um therefore made the training behave the way it did. Uh and that is one of those cases that you can catch with these hashtags because you know this computation should be the same across all replicas, but it wasn't. Um Which brings me to the next uh instance of that happens at scale. Uh in this case exploding gradients. Um again, we're looking at two different loss curves and the corresponding gradient norm curves. Um, the purple run is our initial training run for Laguna M 1. We're a bit further into training here around 50,000 steps or so. And you can see it stops converging, right? Like it just flattens out. And the reason here was that during training the uh activations grew and grew um right before the um LM head, the unembedding. And um we have to perform some sort of accumulation here uh because we use tensor parallel for the uh unembedding. And that accumulation um was performed in BF16 by default. And because of the growing scale that we observed in the activations, um there wasn't enough um numerical precision available anymore um to do this accurately. And hence the model just couldn't learn anymore. And this is also very uh dramatic point for this to happen because it from there on it really like back propagates into the full model trunk. Um, the orange curve is essentially just adding a fix on that. So, we took the checkpoint from the purple curve. We moved that accumulation into FP32 and from there on the model started converging again. Um, the gradient norm, as you can see, actually started decreasing. Um, before then we had an increasing trend. And um this is also something you can only observe at scale. And that will break your model if you if you're not careful about it. So, as Mara said, we took all of these insights on data, on um numerics, and so on, and we turned them from M 1 into XGen-2. That's why we say it's a new generation model. Um, this included uh increasing diversity, reducing data repetitions, um all of these numerical things I just mentioned, and just adding more observability and checks on that side, as well as generally optimizing the training in the architecture. And XGen-2, if you look at it, it's open weights, right? So, you can download uh you can download and use it for free. Um, it's one of the most competitive models for its size and for coding specifically. That's what we focus on Agentic coding. Um, so we're pretty happy with it. However, you can say that um the model with 33 billion parameters is pretty small. Like you wouldn't probably observe any issues anyways training it. So, what was important to us was to scale this, right? And this is where Laguna S comes in. This model is not public yet, so this is a preview. Um as I said, we treated XS as a test bed in a sense, and uh with Laguna S we scaled this to a model that's 118 billion uh total parameters and 8B active parameters. Again, we trained it on 30 trillion tokens on 4,000 GPUs. So, the scale was sufficient to not only test whether uh all the improvements we made on the data and architectural side had hold, but also if any of these numerical issues come up again. And um of course, something happened. Um in this case, it doesn't actually have anything to do with scale, so it was just unfortunate. Um in this case, we had a race condition because we added FP8 training uh based on Deep Chem FP8 kernels that are also like open source. Um we noticed these because we hit illegal memory accesses as well as NaNs in the gradients, uh which after a while of debugging we traced back to those kernels. Um there's also an unobservable unobservable effect um that you wouldn't know about if you don't know that there's an issue. Uh in our case, we noticed about 0.5% of the gradient gets silently corrupted, essentially replaced by random values. Um we do have a fix available that's in a PR right now. Uh it's not been merged to Deep Chem yet, but it's public on that QR code if anyone is interested. Um and it's also an interesting point because it's a blind check and a blind spot in the hash checks. In real training runs, you don't have any redundancy where you have the same model weights and the same data, so you can never check if forward and backward actually behave the same across different model replicas. So, you can also never check if there's a race condition in that. Uh that's something that we're working on right now uh to essentially have a hash checker um that can also do that as a at least as a dry run. Um and I want to end on um some early results from uh this new model um and demonstrating how it performs against some open weight models and also against our previous models. So, first I want to caveat this with these are base model evals, right? They are partly indicative of how the final model will look, but also not perfectly, right? Um there's still post training happening. Uh not all of these will translate one-to-one to the final model. Um but if we look at them uh specifically on the coding part of of the E valves, so for instance multiple E love code bench, big code bench, uh Laguna S is not only stronger than access to two, which is our previous uh smaller model that performed very well, but also than the much larger M dot one. Um and it's also much better than GLM 4.5 air, which is admittedly a bit older, and then we turn 360, which is quite recent, and then deep seek V4 flash max, which is quite recent and a fair bit larger. Um we can see it's competitive on big bench hard for instance. It doesn't achieve the the top E valve results compared to these models, but it's quite close. Um we also see it's quite close on E valve plus, and uh quite importantly for us, it does very well on sweep bench agent less multilingual, um which we use to sort of proxy agentic performance during pre-training. Um and in that case it performs much better than all the other all other models we tested here. Um I also want to point out that of course there are like it's not the strongest model in the world, right? Like for instance MMLU pro knowledge benchmark is something we don't care about that much compared to coding because we want to build the strongest agentic coding models. Um so here like compared to Nema tron and deep seek uh we have to say that they perform much better, and this mainly comes down due to data, right? It's a data gap um that we could plug if we wanted to. Um but I think the the point is all of the things we we found before were included in the recipe. The recipe held, it scaled, and we will continue scaling it from here. So this model will also be available sometime in the future relatively soon. Um again open weights, so all of you can download it and use it for free. And um with that I want to thank everyone for attending our talk. Um I added also a link to our careers page and our Twitter page if you want to uh check it out. And yeah, thank you very much. \>\> \[music\] \>\> Hey.