--- schema: 1 type: video title: >- The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside description: >- Good code data runs out, so poolside manufactures more of it, and the hard part is making it teach. Their synthetic pipeline pairs templates with supplementary context and spreads generations across an axis of phrasing, with difficulty tuned so a task is neither trivial nor so hard the model learns nothing from it. Multistage pipelines port existing data into new shapes, swapping character styles or plots and turning single prompts into multi turn chats, while an orchestrator polices every generation and drops the ones that miss. On the training side the team trusts nothing: run two replicas of the same model on the same data and they must return the same number, or the run gets killed. That is how the messy failures surface. Broken GPUs show up as a spiky loss curve, a numerical precision bug in tensor parallel accumulation quietly flattened another until they patched it, and silently corrupted gradients from a race condition were a blind spot nothing caught. The payoff is a 118 billion parameter model built for agentic coding whose early results already edge out GLM 4.5 Air, on a recipe that held as it scaled. Speaker info: Marah Abdin, poolside: - https://x.com/marah_i_abdin - https://www.linkedin.com/in/marah-abdin - https://marahabdin.com Robert McHardy, poolside: - https://x.com/robert_mchardy - https://www.linkedin.com/in/robert-mchardy - https://www.robertmchardy.de Timestamps: 0:00 - Introduction: synthetic data and pre-training at poolside 1:52 - Why synthetic data 3:11 - Limitations and the training budget 4:44 - Inside the synthetic data pipeline 6:37 - Multistage pipelines and porting data 7:43 - Multi turn chats and policing generations 9:03 - Pre-training: trust nothing, crash on mismatch 10:41 - Failures at scale: broken GPUs 11:56 - Numerical precision and corrupted gradients 13:15 - A 118B model for agentic coding 15:07 - Early results vs GLM 4.5 Air canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/the-messy-reality-of-scale-synthetic-data-and-pre-t-f70cf967d8f6 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/the-messy-reality-of-scale-synthetic-data-and-pre-t-f70cf967d8f6.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T06:30:10.505Z' language: null children: [] videoId: KhYifX22yhE durationSeconds: 1051 publishedAt: '2026-07-26T01:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 124 endSeconds: 201 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=124' heldReason: rights - ordinal: 1 text: null startSeconds: 124 endSeconds: 201 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=124' heldReason: rights - ordinal: 2 text: null startSeconds: 212 endSeconds: 288 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=212' heldReason: rights - ordinal: 3 text: null startSeconds: 212 endSeconds: 288 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=212' heldReason: rights - ordinal: 4 text: null startSeconds: 300 endSeconds: 403 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=300' heldReason: rights - ordinal: 5 text: null startSeconds: 300 endSeconds: 403 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=300' heldReason: rights - ordinal: 6 text: null startSeconds: 415 endSeconds: 469 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=415' heldReason: rights - ordinal: 7 text: null startSeconds: 415 endSeconds: 469 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=415' heldReason: rights - ordinal: 8 text: null startSeconds: 415 endSeconds: 469 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=415' heldReason: rights - ordinal: 9 text: null startSeconds: 481 endSeconds: 545 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=481' heldReason: rights - ordinal: 10 text: null startSeconds: 556 endSeconds: 643 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=556' heldReason: rights - ordinal: 11 text: null startSeconds: 643 endSeconds: 655 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=643' heldReason: rights - ordinal: 12 text: null startSeconds: 720 endSeconds: 732 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=720' heldReason: rights - ordinal: 13 text: null startSeconds: 808 endSeconds: 909 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=808' heldReason: rights - ordinal: 14 text: null startSeconds: 921 endSeconds: 1051 watch: 'https://liquidgold.cc/watch/KhYifX22yhE?t=921' heldReason: rights --- # The Messy Reality of Scale: Synthetic Data and Pre-Training — Marah Abdin & Robert McHardy, poolside Good code data runs out, so poolside manufactures more of it, and the hard part is making it teach. Their synthetic pipeline pairs templates with supplementary context and spreads generations across an axis of phrasing, with difficulty tuned so a task is neither trivial nor so hard the model learns nothing from it. Multistage pipelines port existing data into new shapes, swapping character styles or plots and turning single prompts into multi turn chats, while an orchestrator polices every generation and drops the ones that miss. On the training side the team trusts nothing: run two replicas of the same model on the same data and they must return the same number, or the run gets killed. That is how the messy failures surface. Broken GPUs show up as a spiky loss curve, a numerical precision bug in tensor parallel accumulation quietly flattened another until they patched it, and silently corrupted gradients from a race condition were a blind spot nothing caught. The payoff is a 118 billion parameter model built for agentic coding whose early results already edge out GLM 4.5 Air, on a recipe that held as it scaled. Speaker info: Marah Abdin, poolside: - https://x.com/marah_i_abdin - https://www.linkedin.com/in/marah-abdin - https://marahabdin.com Robert McHardy, poolside: - https://x.com/robert_mchardy - https://www.linkedin.com/in/robert-mchardy - https://www.robertmchardy.de Timestamps: 0:00 - Introduction: synthetic data and pre-training at poolside 1:52 - Why synthetic data 3:11 - Limitations and the training budget 4:44 - Inside the synthetic data pipeline 6:37 - Multistage pipelines and porting data 7:43 - Multi turn chats and policing generations 9:03 - Pre-training: trust nothing, crash on mismatch 10:41 - Failures at scale: broken GPUs 11:56 - Numerical precision and corrupted gradients 13:15 - A 118B model for agentic coding 15:07 - Early results vs GLM 4.5 Air Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/the-messy-reality-of-scale-synthetic-data-and-pre-t-f70cf967d8f6 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.