Talk by Ari Morcos (DatologyAI) on data quality as the "compute multiplier" in AI training: trading data quality for compute on the scaling curve, treating data pipelines like an oil refinery (cleaning, curating, creating, composing data; quality classifiers, deduplication, synthetic generation, and stage sequencing), signal per token as the scarce resource, and claimed outcomes such as small multilingual models beating larger ones, inference efficiency, and customer results (Thomson Reuters, Arcee Trinity). Closing claim: manufacturing high-quality data can be cheaper than buying more compute. Timestamps cover scarcity of good data, curation, model comparisons, synthetic data, and case results. # Channel Identity Observations — "Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI" ## Speaker & Platform Framing - The talk is delivered by **Ari Morcos (transcribed "Marcos"), CEO and co-founder of DatologyAI**, opening a "data quality track" at what appears to be a conference. > "My name is Ari Marcos. I'm the CEO and co-founder of Datlogy AI... really excited to kick off the data quality track today." - The speaker explicitly positions the company's identity around data: > "data quality is what we live and breathe at Datlogy... the company's name literally means the science or study of data." ## Explicit Self-Description (Creator Claims) - **Company scope claim**: DatologyAI is described as a data-curation company, not a data source: > "You can think of us as the oil refinery for data. We don't source new tokens like many data providers. Rather we take existing tokens... and make them way better." - **Methodology claim** — the "four C's": **clean, curate, create, compose**, covering heuristic filtering, benchmark decontamination, quality classifiers, redundancy reduction, up/downsampling, task distribution matching, and curriculum composition. - **Research claim**: "about half of our team at Datlogy are just researchers," with foundational work in "Beyond Scaling Laws" ("best paper at NeurIPS") and multiple public research releases claimed. - **Hiring/lead-generation claim**: the talk closes with "we're hiring for a bunch of different roles" and an invitation to chat about building/customizing models. ## Intended Audience - **Technical ML practitioners and decision-makers**: references assume familiarity with H100 pricing, test-time compute, reasoning models, scaling laws, benchmark decontamination ("As I'm sure you all know, benchmaxing has become a real problem"), N-gram matching, and Pareto frontiers. - **Organizations wanting custom models** are directly addressed: > "Fundamentally we want to help folks build models that wouldn't be able to do so otherwise." - **Cost-sensitive buyers**: the RCI/Trendy Large case argues a frontier-competitive model can be built "for less than $20 million total," contrasting with "hundreds of millions of dollars." ## Claimed Scope Demonstrated in Content - **Vision-language models**: claims ~14 percentage point improvement from curation alone; matching Qwen 3.5 4B performance "with about 145x less training compute." - **Multilingual models**: curating English data shown to transfer to non-English performance; "a little better than Qwen 3 while having roughly 8x less compute budget." - **Synthetic data**: a "rephrasing" platform the team claims to have "pioneered several years ago" and now "table stakes." - **Customer case studies**: Thomson Reuters (legal mid-training; post-training gains "almost tripled") and RCI (17T-token open model, "fully open, US-made," "no closed model usage"). ## Observable Presentation Patterns (Demonstrated, Not Claimed) - **Thesis-driven structure**: a single repeating argument — "Data quality is a compute multiplier" — stated and restated with supporting charts (scaling plots, Pareto frontiers). - **Vendor keynote framing**: the talk blends technical tutorial content with product positioning and named customer wins, ending in a hiring/contact slide — demonstrating this channel hosts **company-branded technical talks**. - **Quantitative evidence style**: consistent use of concrete numbers (40% H100 price rise, 8x token usage in reasoning models, 100B mid-training tokens "<1% of the pre-training budget," "$20 million total"). - **Forward references within the event**: mentions an upcoming talk ("you'll actually hear from Verun, who was the pre-training lead... later today"), indicating a multi-speaker curated track. ## Boundary Notes - All results (benchmark gains, compute multipliers, customer outcomes) are **creator claims presented on the speaker's own slides**, not independently verified in the material. - No private motivations are stated; commercial interest is, however, explicitly demonstrated through the customer case studies and closing hiring/contact appeal. # Delivery Observations — Ari Morcos, "Data Quality Is the Compute Multiplier" ## Tone and Register - **Conversational-but-technical conference talk style**: opener is casual ("Good morning everybody... really excited to kick off the data quality track") while the body uses dense technical vocabulary ("marginal information gain per data point," "pareto frontier," "log flops"). - **Promotional framing** consistent with a founder/CEO keynote: repeated company references ("what we do at here," "what we work on at Datology") and a closing hiring pitch ("we're hiring for a bunch of different roles"). ## Signature Phrases and Repetition - **Central recurring phrase, explicitly flagged by the speaker:** > "you're going to hear me say this over and over again. Data quality is a compute multiplier" The phrase **"compute multiplier"** recurs at least three more times ("really shows this compute multiplier point," "You can again see that we get the same sort of compute multiplier effect," "the single most underleveraged compute multiplier"). - **"Table stakes" used twice** for contrasting purposes — first dismissively about cleaning ("That's kind of basic table stakes") and later approvingly about rephrasing ("has now become basically table stakes for building a very strong model"). ## Teaching Habits and Rhetorical Patterns - **Rhetorical question → answer structure** to transition between sections: > "So in a world where compute is increasingly scarce and you need to make models better. Well, what do you do? Well, we work on data." > "So then how do you make it better?" > "What about text models?" - **Mnemonic device**: the "four C's" — "clean, curate, create, and compose" — as an organizing framework. - **Analogy use**: "You can think of us as the oil refinery for data." - **Concrete numbers to anchor claims**: "40% up from their lows," "eight times as many tokens," "14 absolute percentage point improvement," "145x less training compute," "less than $20 million total." ## Vocabulary and Content Markers - Frequent **quantifier-heavy language**: "10 times," "50x more training compute," "two to three times more effective," "5x again." - **Hedged speculation when speculating**: "it's not implausible today that we might see access to some of the frontier APIs actually get limited." - **Direct audience address and assumed shared knowledge**: "As I'm sure you all know," "I'm sure that many of you have seen or experienced," "For those that are a little bit more technically." ## Fluency Markers - **Heavy filler usage** throughout: "um" and "uh" appear very frequently, sometimes clustered ("um, we uh do all of our own research"). - **Self-corrections and restarts**, e.g., "now you can feed the model into the data, but it's still sorry, feed the data into the model," and "your text your text your text model." - **Repeated words for emphasis**: "it's is" in the opening, "well well beyond uh that frontier," "to to to achieve." ## Structure and Pacing Signals - **Explicit time management**: "let me spend the last few minutes just quickly talking," "in the last minute or so I just want to quickly talk about." - **Verbal signposting of slides/plots**: "So what's plotted here is...," "what you can see is," used repeatedly when walking through figures. - **Forward references to other talks** (track-host habit): "you'll actually hear from Verun... later today... so look forward to that talk"; "I think we'll hear a lot about the synthetic data in various forms throughout the day." ## Confidence vs. Uncertainty - Claims about **his own company's results** are stated flatly and with numbers ("almost tripled," "about five percentage points"). - Claims about **industry trends** carry explicit hedges ("it's not implausible," "might be a world where"). - One **personal-opinion statement** is offered as such: "I don't want to live in a future where only developed countries can access this very effectively." ## Uncertainty Notes - Timestamps are bracketed line markers, not verified media timecodes; delivery pace, gestures, and visuals are not observable from the transcript alone. - Whether the filler words reflect live nervousness or transcription artifacts cannot be distinguished from text alone. # Observation Document: "Data Quality Is the Compute Multiplier" — Ari Morcos, DatologyAI ## Speaker - **Ari Morcos** (transcript rendered as "Ari Marcos"), CEO and co-founder of **Datology AI**. All positions below are his, expressed in a talk opening a data quality track. ## Core Thesis - **Data quality is a compute multiplier**: improving data quality makes the learning curve steeper — better performance at the same compute budget, or the same performance at much lower compute. - Framing: "It's almost always better to repeat high-quality data than to show low-quality data... up to a threshold." ## Contextual Claims: Compute Scarcity - H100 prices rose ~40% from end-of-last-year lows, reversing a multi-year decline. - Reasoning models use **8x as many tokens** as non-reasoning models; projected to 5x again within roughly a year. - Frontier API access could plausibly be limited by **inference constraints**, not business decisions: - Google capped Meta's Gemini usage (cited as example). - OpenAI selling "token futures" / guaranteed capacity. ## Principles of Data Curation (Stated) - Goal: **maximize marginal information gain per data point** — maximum signal per token/batch. - There is **no single "golden dataset"** — optimality is task-relative (legal vs. healthcare example). - Data must be **diverse**, **informative**, and correctly **mixed**; insufficient diversity causes brittleness. - Datology's "four C's" pipeline: **Clean, Curate, Create, Compose**: - Clean: heuristic filters (e.g., Gopher-style), benchmark **decontamination** via low-threshold n-gram matching. - Curate: quality classifiers, topic taxonomy balancing, redundancy reduction, up/downsampling, task distribution matching. - Create: synthetic data via **rephrasing**. - Compose: sequencing data across training stages; exploratory "continuous curricula." - Positioning: Datology as an "oil refinery" — refines existing tokens rather than sourcing new ones. ## Research Results Claimed ### Foundational work - "Beyond Scaling Laws" (Morcos at Meta, NeurIPS best paper): choosing data correctly can **change the exponent** of scaling laws. ### Vision-language models (Mammoth dataset, ~25B tokens, fusion adapter training) - ~**14 absolute percentage point** improvement from curation alone, other factors held constant. - Matched Qwen 3.5 4B performance with ~**145x less training compute** (no post-training). - Curated-data models gave **more concise answers**; similar performance with ~35x fewer flops per correct answer. ### Multilingual text models - Used only **8% multilingual tokens** (most languages ≤6B tokens). - Slightly better than Qwen 3 with ~**8x less compute**. - Cross-lingual transfer: **curating English improves non-English performance**; transfer magnitude correlates with language similarity to English; smaller reverse effect also observed. - Scaling predictability: two small dense models' trend line ran through the much larger RCI "Trendy Large" (17T tokens) despite ~50x compute difference — small curated runs can **de-risk large training runs**. ### Synthetic data ("Beyond Web") - **Rephrasing** (pioneered by his team, now "table stakes"): transform a source document into many formats (e.g., true/false questions). - Claims: no model collapse risk since information derives from the source; the trained model can exceed the rephrasing model's quality; **what you rephrase matters** — random selection underperforms. ## Customer Case Studies ### Thomson Reuters - 100B-token mid-training (<1% of pre-training budget) on open-source model: legal capabilities +~5 points (LegalBench), **no catastrophic forgetting** — general capabilities also rose. - Key qualification: majority of mid-training data was **representative of the pre-training distribution**, not domain-specific. - Post-training gains **nearly tripled** when applied to the mid-trained model vs. default instruction-tuned model. - Inferred principle: training stages should be treated **synergistically**, not as independent hand-offs. ### RCI (Trendy Large) - Fully open, US-made model; 17T curated public tokens; no proprietary data; **no closed-model usage**. - Competitive with open frontier; matches GLM5 and Kimi on many tasks; outperforms Claude on a couple. - Total cost (compute, salaries, R&D, several models): **under $20 million**. - Claim: customizing models does **not** require hundreds of millions; a narrow-domain model is feasible for "high six figures" million dollars; Trendy Large represents the general-purpose upper bound. ## Stated Caveats & Qualifications - Repetition of high-quality data helps only "up to a threshold." - Synthetic data rephrasing fails if documents are selected randomly. - Preventing forgetting requires mostly pre-training-distribution data — a stated necessity, not optional. - Multilingual results rest on small non-English token budgets (an observation, not a limitation he flags as a weakness). ## Closing Positions - Data quality is "the single most underleveraged compute multiplier." - The problem is a **frontier research + engineering** challenge: scoring data across many axes at petabyte scale. # Speaker Attribution Record — Video `01a15ca8-07bf-7249-823c-478874ee429f` ## Video - **Title:** Data Quality Is the Compute Multiplier — Ari Morcos, DatologyAI ## Speakers ### Ari Morcos (transcribed as "Ari Marcos") — **Creator / Sole Speaker** - Self-identified: **CEO and co-founder of Datology AI** (transcribed "Datlogy AI"). - Introduces himself as kicking off the **data quality track**; delivers the full talk. ### Referenced (non-speaking) individuals - **Verun** (spelling per transcript) — pre-training lead for the RCI "Trendy Large" model; mentioned as speaking **later today** at the same event. - Speakers referenced indirectly as part of the track's "amazing lineup of talks." ## Referenced Sources and Third-Party Claims ### Company/own research - **"Beyond Scaling Laws"** — paper authored by the speaker at Meta, described as foundational to Datology; claimed **best paper at NeurIPS** ("nurips" in transcript) a couple of years prior. - Datology's public research releases; claimed team is ~half researchers. ### Third-party claims (attributed to speaker, not independently verified) - **H100 prices** ~40% up from end-of-last-year lows. - **Reasoning models** use ~8x tokens vs. non-reasoning; projected to 5x again within a year. - **Google capped Meta's Gemini usage** due to inference constraints. - **OpenAI** selling "token futures" / guaranteed token capacity. - Model comparisons citing **Qwen 3 / Qwen 3.5, InternVL, GLM5, Kimmy, Claude, Llama-style models, Liquid models, TinyCohere** (transcript spellings). - Benchmarks referenced: **Multilingual MMLU, LegalBench, Gopher-style heuristic filters, "mammoth" dataset (~25B tokens), Pareto frontier comparisons**. - Customer case studies: **Thomson Reuters** (legal mid-training/post-training), **RCI** ("Trendy Large," 17T tokens, <$20M total cost claim, "US-made, fully open," no proprietary data or closed-model usage). ## Promotional Segments ⚠️ - **Hiring pitch:** speaker closes by noting "we're hiring for a bunch of different roles." - **Product/service promotion throughout:** talk promotes Datology's curation services ("four C's," "Beyond Web" synthetic data platform, customer results) — company-positioned content presented as a talk. - Invitation to chat for model-building/data services at close. ## Availability Limits - **Comments availability:** not stated in transcript or metadata. - **No curated metadata supplied** beyond title; speaker/company names rely on transcript (with transcription variants noted above). - Speaker notes the field has **little published research** due to competitive disincentives — a provenance limitation he explicitly states. Good morning everybody. My name is Ari Marcos. I'm the CEO and co-founder of Datlogy AI. Uh and really excited to kick off the data quality track uh today. Uh data quality is is what we live and breathe uh at Datlogy. It's all we think about. In fact, the company's name literally means the science or study of data. Um, so very excited to see the increasing excitement and interest in this area and a amazing lineup of talks uh today. So today I'm going to tell you about why data quality is a compute multiplier that we're all overlooking and where we can make massive gains by just working on better data. Um, we've seen that compute availability over the last six months has become extremely scarce and is only getting worse. We saw H100 prices reverse their several year-long drop which is normal for hardware and all of a sudden come up where now they're about 40% up from their lows um at the end of last year. Um as test time compute has become a critical part of models and as we have put more and more thinking tokens in we're seeing that the number the token usage is absolutely skyrocketing. Um reasoning models use eight times as many tokens um as non-reasoning models and that's projected to 5x again in the next year or so. Um, so the number of tokens we're pushing through goes higher and higher and that constrains compute even further. And this has led actually to a world where it's not implausible today that we might see access to some of the frontier APIs actually get limited or go away. Um, as one example, Google just capped uh Meta's Gemini usage because of inference constraints. OpenAI has effectively started selling token futures or you can guarantee token capacity some amount of time into the future. This is only necessary because they are legitimately wondering there might be a world where access to frontier API tokens is limited not as a business decision but because there's just simply not enough inference and first party products will be prioritized. So in a world where compute is increasingly scarce and you need to make models better. Well, what do you do? Well, we work on data. Um I you're going to hear me say this over and over again. Data quality is a compute multiplier because what it does is it it makes the learning curve steeper. So this is just a very simple schematic of performance on the y-axis as a function of data on the x-axis. Note that this x-axis, you could swap it out for data, for compute, for time, for dollars. They're all the same x-axis fundamentally. Um, and if you can make data quality better, you can turn this gray curve um into this blue curve. And that means that now you can get dramatically better performance for the same compute budget as if you had trained with far more compute. Um and similarly you can get the same performance for a much smaller compute budget. Um which is exactly showing you how you can get performance as if you had spent 10 times this 10 times as much on compute and really shows this compute multiplier point. Um so how do you actually do this? Well fundamentally the idea is we want to make it so that we get the maximum signal per token and per batch. Um for those that are a little bit more technically what we want to want to do here is maximize the marginal information gain per data point when we show it to the model. what data is going to teach the model the most. Um, and that's all about finding data that's relevant to the use cases that you want. One thing that you know is very important is that there's no one golden data set to rule them all that's good for everything no matter what you want to do. A data set's only going to be optimal with respect to a particular set of output tasks that you want the model to do. So if you want a great legal model, you're going to want legal data more than healthcare data and vice versa. It needs to be diverse. A lot of the issues we see with model robustness and brittleleness comes from try training on data that's not diverse enough. So the model can answer a question correctly if it's presented just so but if it's presented a little bit differently now everything breaks um needs to be information and you have to mix the data correctly. This is a hugely difficult part of you now have many different sources. How do you combine them to actually drive the largest improvement in performance? So um this is a high level of what we do at here. Um you can think of us as the oil refinery for data. We don't source new tokens like many data providers. Rather we take existing tokens coming from public data sets, proprietary data sets and licensed data sets and make them way better. And how do we do that? Well, we do that through these four C's. Um, clean, curate, create, and compose. Um, so cleaning is fairly straightforward. This is doing things like heruristic filters, all a gopher and things like that. Removing documents that have, you know, only 10 characters in them or all winging. That's kind of basic table stakes. Um, benchmark decontamination is incredibly important. As I'm sure you all know, benchmaxing has become a real problem and makes it very difficult to interpret model results. So, we rigorously decontaminate all of our training data with respect to all downstream benchmarks with a pretty low engram to ensure that that's the case. That gets you to a point where now you can feed the model into the data, but it's still sorry, feed the data into the model, but it's still not very good. So then how do you make it better? Well, it's a combination of many things ranging from quality classifiers and tonomy across different topics and balancing that redundancy reduction. So removing data points that are not the same that are semantically similar but convey very similar information even if they're not the same pixels themselves say upsampling and downsampling data points based off the quality and the relevance and then task distribution matching identifying what data do you actually need in order to solve this given task. That now gives you a data set that is very high quality but is typically still too small. And that's where synthetic data comes in. Now we can go and rephrase that data as effectively a very fancy form of data augmentation to produce dramatically more data in many different formats and this helps a lot both with kind of uh data size and with diversity um because we can really inject a lot of diversity in through this and then finally you know have these data sets how do you combine them and how do you com and how do you sequence them across different training stages it's now become table stakes that any large model is generally trained for at least three phases of data um how do you do that um and can you actually even do continuous uh curricula and things like that which is a lot of what we work on at Dtology. Um and that ultimately gets you a much better data set out. All right, so that's a high level of kind of what we need to do. What can you actually get out of this? Can this actually really make a massive difference? Um so we've about half of our team at Datlogy are just researchers. Um and you know we uh do all of our own research on how we do data curation effectively. um because this is such a critical part of the uh model building pipeline uh there's very little published here um because there's a very strong disincentive not to share how you do this um the kind of foundational paper for daty was one I wrote when I was at meta called beyond scaling laws um which was fortunate to get a best paper at nurips a couple years ago which showed that if you choose your data correctly you can actually bend the scaling laws itself you can change the exponent um and that's because you're now not wasting your time looking at redundant or unnecessary data That was very much the proof of principle for all of Dtology. And we've since expanded this into many public research releases. We've shared of various ways to improve models just through data curation. I'm going to go through a couple of those results now um and show you what we've been able to to to achieve. Um so first let's talk about vision language models. How can we improve um VLMs just through data curation alone? Um so in this case what we did is we took the mammoth data set. This is a fairly small data set about 25 billion um tokens uh that we use for the purposes of training the fusion adapter layer between your um text uh your text your text model and your vision model. Um and what you can see this is a scaling plot where we have error on the y-axis as a function of log flops um on the x-axis. Um so there's about a scale of a thousand from the left uh most part of this plot to the rightmost part. Um and what you can see is that if you look at the parto frontier defined by many of the best uh public VLMs like the Quen 3 series and 3.5 intern VL etc you can see that a model trained on daties data are able to go well well beyond uh that frontier and I'll note this is actually without any post- training as well. Uh so you can get very strong performance across many different benchmarks. Um so just looking at kind of taking the input data set that gray diamond there that's the input data set we use to do our curation. Um you can see that just through curation you're able to get around a 14 absolute percentage point improvement um holding everything else constant just through better data alone. Um and not only that you can also see that we can roughly match the performance of quen 3.54b come with about a percentage of it um while using 145x less training compute in a world with no with less compute. How do you do more? You make data better and now it's as if you had 100 times um the compute. Um, interestingly I mentioned that reasoning models are be are using tokens at a very high rate as well. Well, another thing that we found is that data curation can also lead uh to more concise answers um depending on how you represent the data. So what's plotted here is the mean number of tokens per response um across all the same set of models for the large most part um that we just showed. Um and you can see that that models trained on data those three blue lines right at the top are all extremely concise. Um and if we do kind of the same sort of plot um but now on the x-axis instead of log training flops this is now um log flops per response. Um so this is inference efficiency. Um you can still see that we go well on the ex by beating that router frontier um and you know roughly get uh similar performance to quen 35 with 35 fewer uh times fewer flops uh per correct answer. So data curation can make a huge impact in VLMs. What about text models? Um, one of the most uh challenging things about many models is that they work very well on English data, but they don't work well um for non-western use cases in general. The internet is an extremely biased view of the world that does not represent the world uniformly at all. And this has major implications for fairness and for the usability of these models across the world. I don't want to live in a future where only developed countries can access this very effectively. Um, so how do you do this? Well, curation again can be a massive lever here. Um so what I'm plotting here now is a similar plot error on the y- axis as a function of log flops about 100x going from left to right here. Um this is highlighting multilingual MMLU performance. You can see we have a parto frontier here defined by many models. The quen models um the liquid some of the liquid models. Um the green square is tiny coher's best multilingual model. You can see again that we're well off the predto frontier with a couple things I really want to highlight. First off we only use 8% of the data here as multilingual tokens. Um, so most languages actually only had um at max 6 billion tokens here. So these are not massive amounts of data uh in the non-English languages that are going in here. Um you can again see we get the same sort of compute multiplier effect. We're a little better than Quen 3 um while while having roughly 8x less compute budget um here. So you can make a huge improvement. One last thing I want to show here is that if you look at the two blue points on the upper left here um those are both dense llama style models trained for a trillion tokens um on curated data. The point on the lower right here I'll come back to but is a model trained by one of our customers rci trendy large that was trained on 17 trillion tokens and is a hypersparse. Um and what you can see is that if you take the line defined by the two smaller models, um it goes mostly right through that blue star uh which is training large despite it being trained um with 50x more training compute. So if you use your data correctly and you simulate token scarcity appropriately, you can also get um very predictable scaling to much larger models and you can derisk a run with 50 or 100 times less compute effectively um before you actually go and scale up the hero run um and find that maybe it doesn't end up where you want it to be. Um an interesting scientific result I want to share here is that we also see very strong cross-lingual benefits um from curation. Um so what's plotted here is the non-English accuracy um where in the the the left bar is showing not curating anything at all and then the right bar just curating the English. Um we also curate all the non-English data and that leads to much better performance. But I just want to show this because I think it's quite interesting that curating English data benefits non-English performance. Um and that's because we see this cross-lingual transfer where the model understands how English relates to say Spanish and and so therefore making it better at English would also make it better at Spanish to some extent and interestingly we see that the the um the size of magnitude of that transfer um is strongly correlated uh with the similarity between English and that language um and we also see it go the other way although the effects a bit smaller um where curating the non-English data also helps to benefit English data um performance um okay And let me talk a little bit about synthetic data. Um we take an approach to synthetic data that we call rephrasing. This is something that our team pioneered several years ago and has now become um basically table stakes for building a very strong model. Um in anyway I think we'll hear a lot about the synthetic data in various forms uh throughout the day. Um but fundamentally with beyond web our goals is how can we define a synthetic data platform that works extremely well um and can be applied to anyone's proprietary doc data and documents. Fundamentally we want to help folks build models that wouldn't be able to do so otherwise. And that's what where data quality can make an absolute difference. Um so to give an example, we might take a document like this about a corporate takeover and we might convert that into one of hundreds of templates. One of which might be a series of true false questions. Um by doing this, there's a couple things that are really great. Number one, because all the information is coming from the document on the left, you don't have any issue with model collapse. Um and you can actually train models um that are much better than the rephrasing model because the rephrasing model doesn't actually have to teach and understand all the concepts. All it needs to do is transform the left document into a true false questions accurately which is a much easier task. And then you do this into many many different formats um throughout data. This effectively increases diversity and it makes it so you learn a lot more from the highest quality data points. One thing that's really critical here, what do you rephrase? All documents are not created equal for rephrasing. If you just pick random sets of documents to rephrase, you will not get a great result. Um but if you find the high quality documents and rephrase them, it can make a big difference. Um and we've seen if you compare this to lots of other public synthetic corpora, we can get much better performance much faster. Um and critically this can be applied to any any proprietary data um in one of our customers own environments. All right. Uh let me spend the last few minutes just quickly talking about a couple um of what we've seen uh the things we've seen with our customers where this can actually drive really gains. Um so first one of our customers Thompson Reuters um has really focused on post training quite a bit. um and they have a very sophisticated post- trainining infrastructure with the goal of uh building better legal models on their proprietary high-quality legal data that they have. So we partnered with them to mid-train a model first on a combination of their data um and uh public data um to then make much better legal reasoning models. So what do we see? Well, first off, look at the left here. Um in this case, we took a an open source model um and then uh just did continued pre-trainer and mid-training on a 100 billion tokens. What you'll see here is that we see that legal capabilities go up about five percentage points um as measured by uh legal bench after you do this 100 billion mid-training which was less than 1% of the pre-training budget. Um but you don't get catastrophic forgetting. You also see the general capabilities go up as well. I'm sure that many of you have seen or experienced um when you try to adapt a model to a particular domain, you lose general performance. Not if you use the data correctly. Um, the key here is actually the majority of the data we showed the model was actually data that was representative of the pre-training distribution. There was only some of the domain specific data in and that's necessary to prevent the model from losing the capabilities that it had before. Um, and you can solve this entirely through better data. Um, but this actually isn't the most exciting part here. Um, as I mentioned, the TR team had done a lot of work on post- training. Um, and had a very sophisticated post-training harness. Well, they then applied that to the mid-train model versus just the default instruction tune model. And what they found was that the gain deriving from post-training. So the y- axis here is a delta um as a result of post-training um almost tripled uh when you applied it to the mid-train model versus to the um just default instruction tune model. And that's because its policy um when it starts is now much more accurate um and it can make much better inference. Um, so you can even even if you don't change the post- training data at all, showing your model better domain specific data can actually make post- training two to three times more effective out of the box. Um, which I think really goes to show not only how important data can be in these uh factors, but it also actually goes to show how we really should be thinking about all these stages synergistically rather than as uh three completely independent stages of pre-training and then I hand it off to somebody else who mid-trains who then I hand off to somebody else um who post trains. Um okay uh in the last minute or so I just want to quickly talk about um one other one which is rci who I mentioned that large model trendy large um you'll actually hear from verun who was the pre-training lead for this model later today um so look forward to that talk um but in this case they trained a model this is a fully open um US-made model um uh on 17 trillion tokens that we curated from public uh data sets no proprietary data involved here um and no closed model usage so no asking Claude uh to do this for you Um, and with that, RC was able to train a model that is competitive with the Open Frontier. Um, matches uh, GLM5 and and Kimmy on many tasks. Um, and even outperforms Claude on a couple tasks. Um, but I think what's most exciting about this is that the RC team um, had not trained a model prior to uh, the middle of last year when they started working with us. Um and critically in total across salaries, across compute, across R&D, um across everything for this and several other models, they were able to get to a model that's competitive with the open frontier, um for less than $20 million total. That includes all the repetitions, that includes compute, that includes everything. So if you hear this story over and over again, oh, if I want to customize a model, it's going to cost hundreds of millions of dollars. That's just not true. you can train an immensely powerful model especially in a narrow domain for high six figures million dollars. It's very doable to get a model that's extremely performant. Um this is for a general purpose. So this is kind of the upper bound of that. Um and data quality is how you can do that. All right. So um last slide here just kind of say um summarizing um focus on kind of what's going to give you the most signal per token. That's the thing that matters a lot more than more tokens. that it's almost always better to repeat highquality data than it is to show lowquality data at a certain point um up to a threshold. Um but focus on kind of how can you get that? This is data quality remains the single most underleveraged compute multiplier. If you're sitting in a world where you want to build a model or customize a model and you're limited on compute, how do you get past that? Invest in data and that's something that can do a tremendous amount of effort. Um and then finally, this is a frontier research engineering problem. You need to be able to score and understand data across many different axes. because that's a frontier research problem. And then have that scale up to pabytes of data, massive scale, um, and can be very critical and it can lead to tremendous leverage. Um, and with that, I'll say thank you. Um, I will note that we're hiring for a bunch of different roles on the left here. Um, if you're interested in building or customizing your own model and um, would like to to get much better data out of the box or apply it to your own data, um, would love to chat with you. Um, and uh, thank you very much.