--- schema: 1 type: video title: 'Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI' description: >- The web has roughly 30 trillion tokens of usable text. Frontier pre-training needs more. Here's how DatologyAI generates the rest. Bogdan Gaza, co-founder and CTO of DatologyAI, shares the engineering lessons from running synthetic data jobs at trillion-token scale, including a recent run of about 12 trillion tokens across web, math and code. He covers BeyondWeb, DatologyAI's rephrasing-based synthetic data recipe, and the move from a split Slurm/Kubernetes setup to a single Ray, KubeRay and vLLM pipeline on EKS on HyperPod. He then walks through four bottlenecks: S3 metadata, GPU failures, cross-cluster scheduling and inference tuning. In this talk: • Why seeded rephrasing beats asking a model for synthetic data from scratch • Batching S3 metadata fetches to cut 9–11 days down to about 2 hours • Right-sized partitions plus checkpointing for recoverable GPU failures • Scheduling CPU and GPU together across clusters, and vLLM flag sweeps for about 40% more throughput SPEAKER Bogdan Gaza, Co-founder & CTO, DatologyAI LinkedIn: https://www.linkedin.com/in/bogdangaza/ X: https://x.com/hurrycane LINKS DatologyAI: https://www.datologyai.com BeyondWeb (blog): https://www.datologyai.com/blog/beyondweb BeyondWeb (paper): https://arxiv.org/pdf/2508.10975 CHAPTERS 0:00 Intro 0:12 Synthetic data at trillion-token scale 0:27 Why synthetic data: the data wall 1:47 What DatologyAI does 2:22 BeyondWeb results 4:11 How rephrasing works 5:51 Where we started: Slurm vs Kubernetes 6:31 The stack: Ray, KubeRay, vLLM 7:41 Today's setup on HyperPod 8:40 One workflow: curate, synthesize, train, eval 9:30 Engineering lessons at scale 9:40 Bottleneck 1: S3 metadata (11 days → 2 hours) 12:29 Bottleneck 2: GPU instability and checkpointing 14:59 Bottleneck 3: Cross-infrastructure orchestration 17:53 Bottleneck 4: Inference tuning (+40% throughput) 18:53 Recap 19:13 From 30B to 12T synthetic tokens 20:02 We're hiring Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #SyntheticData #MLInfra #AIEngineer canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/lessons-from-generating-12-trillion-synthetic-token-f7189c9c9718 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/lessons-from-generating-12-trillion-synthetic-token-f7189c9c9718.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:20:10.805Z' language: null children: [] videoId: FQwTqUmcbRg durationSeconds: 1232 publishedAt: '2026-10-02T23:30:27.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 24 endSeconds: 34 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=24' heldReason: rights - ordinal: 1 text: null startSeconds: 46 endSeconds: 113 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=46' heldReason: rights - ordinal: 2 text: null startSeconds: 158 endSeconds: 252 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=158' heldReason: rights - ordinal: 3 text: null startSeconds: 158 endSeconds: 252 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=158' heldReason: rights - ordinal: 4 text: null startSeconds: 252 endSeconds: 262 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=252' heldReason: rights - ordinal: 5 text: null startSeconds: 262 endSeconds: 361 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=262' heldReason: rights - ordinal: 6 text: null startSeconds: 361 endSeconds: 373 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=361' heldReason: rights - ordinal: 7 text: null startSeconds: 396 endSeconds: 407 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=396' heldReason: rights - ordinal: 8 text: null startSeconds: 407 endSeconds: 463 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=407' heldReason: rights - ordinal: 9 text: null startSeconds: 475 endSeconds: 530 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=475' heldReason: rights - ordinal: 10 text: null startSeconds: 530 endSeconds: 542 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=530' heldReason: rights - ordinal: 11 text: null startSeconds: 598 endSeconds: 759 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=598' heldReason: rights - ordinal: 12 text: null startSeconds: 598 endSeconds: 759 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=598' heldReason: rights - ordinal: 13 text: null startSeconds: 769 endSeconds: 901 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=769' heldReason: rights - ordinal: 14 text: null startSeconds: 769 endSeconds: 901 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=769' heldReason: rights - ordinal: 15 text: null startSeconds: 912 endSeconds: 1080 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=912' heldReason: rights - ordinal: 16 text: null startSeconds: 912 endSeconds: 1080 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=912' heldReason: rights - ordinal: 17 text: null startSeconds: 1090 endSeconds: 1134 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=1090' heldReason: rights - ordinal: 18 text: null startSeconds: 1090 endSeconds: 1134 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=1090' heldReason: rights - ordinal: 19 text: null startSeconds: 1157 endSeconds: 1169 watch: 'https://liquidgold.cc/watch/FQwTqUmcbRg?t=1157' heldReason: rights --- # Lessons from Generating 12 Trillion Synthetic Tokens — Bogdan Gaza, DatologyAI The web has roughly 30 trillion tokens of usable text. Frontier pre-training needs more. Here's how DatologyAI generates the rest. Bogdan Gaza, co-founder and CTO of DatologyAI, shares the engineering lessons from running synthetic data jobs at trillion-token scale, including a recent run of about 12 trillion tokens across web, math and code. He covers BeyondWeb, DatologyAI's rephrasing-based synthetic data recipe, and the move from a split Slurm/Kubernetes setup to a single Ray, KubeRay and vLLM pipeline on EKS on HyperPod. He then walks through four bottlenecks: S3 metadata, GPU failures, cross-cluster scheduling and inference tuning. In this talk: • Why seeded rephrasing beats asking a model for synthetic data from scratch • Batching S3 metadata fetches to cut 9–11 days down to about 2 hours • Right-sized partitions plus checkpointing for recoverable GPU failures • Scheduling CPU and GPU together across clusters, and vLLM flag sweeps for about 40% more throughput SPEAKER Bogdan Gaza, Co-founder & CTO, DatologyAI LinkedIn: https://www.linkedin.com/in/bogdangaza/ X: https://x.com/hurrycane LINKS DatologyAI: https://www.datologyai.com BeyondWeb (blog): https://www.datologyai.com/blog/beyondweb BeyondWeb (paper): https://arxiv.org/pdf/2508.10975 CHAPTERS 0:00 Intro 0:12 Synthetic data at trillion-token scale 0:27 Why synthetic data: the data wall 1:47 What DatologyAI does 2:22 BeyondWeb results 4:11 How rephrasing works 5:51 Where we started: Slurm vs Kubernetes 6:31 The stack: Ray, KubeRay, vLLM 7:41 Today's setup on HyperPod 8:40 One workflow: curate, synthesize, train, eval 9:30 Engineering lessons at scale 9:40 Bottleneck 1: S3 metadata (11 days → 2 hours) 12:29 Bottleneck 2: GPU instability and checkpointing 14:59 Bottleneck 3: Cross-infrastructure orchestration 17:53 Bottleneck 4: Inference tuning (+40% throughput) 18:53 Recap 19:13 From 30B to 12T synthetic tokens 20:02 We're hiring Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #SyntheticData #MLInfra #AIEngineer Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/lessons-from-generating-12-trillion-synthetic-token-f7189c9c9718 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.