Mahesh Sathiamoorthy (Bespoke Labs) argues that the hard part of post-training LLMs is not the algorithm but the data and environments feeding it. He covers supervised fine-tuning datasets, frames RL environments as data in a different shape, and discusses why reliability breaks down when agents run autonomously on long tasks. Grounded in the OpenThoughts reasoning dataset, he shares counterintuitive curation lessons — diversity of reasoning traces matters, keeping multiple answers per question helps — plus an example where fine-tuning on correctly tagged data improved credit card compliance metrics, closing with a case for a disciplined curation stack over raw compute. # Channel Presentation Observations — Mahesh Sathiamoorthy, Bespoke Labs ## 1. Speaker Self-Identification and Affiliation - The speaker introduces himself directly: **"I am Mahesh Satyamoorthy. I'm co-founder and CEO at Bespoke Labs. And previously I was a researcher and engineer at Google DeepMind."** - He frames his company's identity explicitly: > "Bespoke is an applied data research lab with a mission to help enterprises and frontier labs access high-quality data and RL environments for their post-training needs." ## 2. Explicit Self-Description of the Channel/Organization's Scope - The talk positions Bespoke Labs' claimed scope as: - **Synthetic data curation** for post-training (SFT): the **Curator** tool, released "last year." - **Reasoning data curation**: Bespoke Stratos, which "eventually formed into the project called Open Thoughts." - **RL environments**: "we do a lot of research and build and ship RL environments" and "we have also been core contributors to Terminal Bench." - **Enterprise post-training services**: "we do a lot of post-training and help enterprises to get their own custom models." ## 3. Intended Audience - The talk is delivered to a **technical AI practitioner/engineer conference audience** (evidenced by references to "last year's AI Engineer conference" and "some of you are excited about… RL environments"). - Secondary audience: **enterprises and frontier labs** — the stated customers in the mission description. - The speaker references fellow presenters ("Nick… also doing something similar," "Ross had a great talk on long horizon"), indicating an in-person conference/talk format. ## 4. Demonstrated Presentation Patterns - **Open-source-first framing**: "the talk will be mostly around open-source work we have done" — demonstrated through extended discussion of Open Thoughts, Bespoke Stratos, Open Thoughts Agents, Curator, and Terminal Bench contributions. - **Research credibility signals**: cites a published paper ("we also published this as a paper in ICLR of this year") and third-party endorsements (a Microsoft CSO tweet; "John Schulman talking about open thoughts… used… internally at Thinking Machines"). - **Practitioner-oriented pedagogy**: walks through a concrete pipeline (source questions → mixing → filtering → teacher-based answer generation → filtering) and shares "counterintuitive" empirical learnings (e.g., "stronger models are not always the better teachers"; "sampling multiple answers per question works pretty well"). - **Enterprise case study style**: a concrete Intuit/Credit Karma deployment example illustrating compliance, latency, and cost outcomes from post-training. - **Ecosystem positioning**: closes with a "reference stack" diagram framing (environments layer, sandbox/orchestration layer, SFT/RL/prompt-optimization layer), aligning his company's work with industry trends. ## 5. Claims vs. Demonstrated Content | Type | Evidence | |---|---| | **Creator claim (mission)** | "help enterprises and frontier labs access high-quality data and RL environments" | | **Creator claim (impact)** | Open Thoughts is "very widely used," citing Schulman and Microsoft CSO | | **Demonstrated (technical depth)** | Detailed curation recipe ablations, scaling-law figures, tag-based data recipe for Credit Karma | | **Demonstrated (opinion/stance)** | "data is the bottleneck" for post-training; "we have moved on from knowing to doing"; SFT "still contributed a lot to the gains" while RL is "very compute-intensive" | ## 6. Tone and Format - Conversational, first-person talk with audience asides ("some of you hopefully know about," "happy to take questions offline"), closing on applause — consistent with a **conference keynote/talk** format rather than a produced YouTube video. - No private motivations inferred; all observations are grounded in the speaker's direct statements and demonstrated talk structure. # Delivery Observations: Mahesh Sathiamoorthy — "Data and Environment Curation for Post-Training LLMs" ## Tone and Overall Style - **Conversational, low-key delivery**: Frequent filler words and hedges throughout — "um," "uh," "you know," "kind of," "what not." > "Um I'm co-founder and CEO at Bespoke Labs." > "you can prompt your way to improving the agents reliability or you can update the harness, you know, the tools and what not." - **Self-positioning as a practitioner sharing open-source work** rather than a polished keynote: > "So very briefly I will tell you a little bit about Bespoke and after that the talk will be mostly around open-source work we have done." ## Recurring Linguistic Habits - **"You know" as a constant connective** — appears dozens of times across the talk, including mid-clause: > "And I will give one concrete example of a post training work we did uh with one of the enterprises." > "as you're curating data, you want to put yourself in the shoes of the researcher to see what what does it take to, you know, uh, actually move the metrics on the models." - **"Kind of" as a softener**, used for both emphasis and hedging: > "post training is a very powerful tool" vs. "It's basically reliability, right?" and "it's kind of beneficial for someone to kind of go do both at the same time." - **"Right?" as a recurring tag question** to check audience alignment: > "We have moved on from knowing to doing, right?" > "for post training, be it SFT or or reinforcement learning, data is the bottleneck, right?" - **"So" as a sentence opener** — begins a large share of sentences: > "So, we we at Bespoke started this effort called Bespoke Stratus..." > "So, this you can read this paper." ## Speech Disfluencies - **Stuttering and restarts**, especially at topic transitions: > "Th- This is the pipeline that we used." > "And and the other thing we do is we we do a lot of post-training" > "it it it's not always compliant" - **Self-corrections**, including one notable name slip: > "Uh, Nick had uh, sorry, uh, Ross had a great talk on long horizon, right?" - **Repeated word duplication** ("the the," "we we," "it it") occurs frequently; possibly a transcription artifact, but consistent with an informal speaking style. ## Teaching and Structural Habits - **Explicit roadmap statements** before diving into sections: > "So, in this talk, I will mostly uh cover some of the work we have done in the open source uh community." - **Signposting transitions** with "the other thing" / "one of the things": > "Um the the other thing I want to kind of mention is..." > "So that's one of the layers. And below that you want various infrastructure..." - **Concrete-example anchoring**: repeatedly grounds abstract claims in named cases (Intuit/Credit Karma, Open Thoughts, Microsoft CSO tweet, John Schulman): > "And I will give one concrete example of a post training work we did uh with one of the enterprises." - **Referring to other talks at the same event** to connect content: > "this again connects back to, uh, what Ross was talking about, where he was talking about algorithms, uh, environments, and compute, right? So, it it feels like, you know, we are kind of converging on something very similar." - **Delegating depth to references** rather than explaining fully: > "But here are some of the learnings... you can read this paper. It has lots and lots of, you know, information about how we did the curation." > "I'm I'm not going to go into details here..." - **Layered/stack explanation** for the closing architecture slide: environments layer, sandbox/compute layer, post-training methods layer — a top-down or bottom-up decomposition pattern. ## Vocabulary and Framing Patterns - **Recipe metaphor** for data curation, used repeatedly: > "We did a lot of work on basically identifying the curation recipe." > "So, after doing all of this, you get the final recipe, right?" - **"Counterintuitive" learnings** flagged explicitly: > "But here are some of the learnings that, you know, some of them are quite counterintuitive." > "This is something that we it's it's kind of counterintuitive." - **Metric/outcome framing**: talks in terms of "moving metrics," "scaling law," "big boost," "improved": > "what does it take to, you know, uh, actually move the metrics on the models." - **Slightly hedged causal claims** — uses "I think" and "probably" when interpreting results: > "The I think the the reasoning is probably that it gives like a variety of how reasoning is done." ## Stable Patterns vs. Isolated Moments - **Stable**: filler-heavy conversational tone, "right?" tags, "So" openers, recipe/scaling-law vocabulary, concrete-example anchoring, signposted structure. - **Isolated**: the Ross/Nick name slip; uncertainty about an audience-familiar method ("I don't know if you if you guys have heard of it"); a moment of apparent uncertainty recalling results ("Um Quinn models were better than, for example, um um Um I'm uh Claude models, I think") — hesitation here suggests recall from memory rather than a stable delivery trait. # Observation Document: Data and Environment Curation for Post-Training LLMs **Speaker:** Mahesh Sathiamoorthy, Co-founder and CEO, Bespoke Labs (self-identified; formerly researcher/engineer at Google DeepMind) ## Background & Company Positioning - Bespoke Labs is an **applied data research lab** helping enterprises and frontier labs access **high-quality data and RL environments** for post-training. - Key projects: - **Curator** — tool for curating synthetic data for SFT - **Bespoke Stratos** — reasoning data curation effort started after DeepSeek's release, which evolved into **Open Thoughts** - Core contributors to **Terminal Bench** - Post-training services for enterprise custom models - **Stated motivation:** A "slight mismatch" exists between people who create data/RL environments and researchers who consume them; he argues doing both simultaneously is beneficial — "as you're curating data, you want to put yourself in the shoes of the researcher." ## Framing Claims - AI evaluation has shifted **"from knowing to doing"** — from knowledge benchmarks (STEM, humanities) to agent benchmarks (SweepBench, TerminalBench). - **Long-duration autonomy** is the goal; what blocks it is **reliability** ("at some point something falls apart... either they call the wrong tool or they made a mistake"). - Levers to improve reliability: prompting, harness updates, **post-training** (his focus). - **Data is the bottleneck** for post-training (SFT or RL), including RL environments, which he frames as "data... in a very different shape." Compute, models, and post-training infrastructure (Fireworks, Tinker, Slime World) are comparatively well-defined. - Post-training has side benefits: reduced latency, improved cost and throughput. ## Open Thoughts (Reasoning Data Curation) - Motivated by lack of high-quality reasoning data outside labs after DeepSeek's release; consortium with Stanford, UC Berkeley, UDub; published as an **ICLR paper**. - Claims a demonstrated **scaling law**: metrics improve as dataset size increases under their recipe. - Cited adoption: Microsoft CSO tweet; **John Schulman** stated he and colleagues at Thinking Machines used it internally. - **Pipeline:** source prompts → mixing (LLM-based quality/hardness checks) → filtering → answer generation with teacher models (DeepSeek, Qwen base, Gemini) → answer filtering; systematic ablations at each stage. ### Stated Learnings (some "counterintuitive") - **Sampling multiple answers per question** (e.g., 16 answers to one question) works better than many questions answered once — his tentative explanation: diversity of reasoning traces helps fine-tuning. - **Stronger models are not always better teachers.** - Synthetic question generation/QA worked; answer filtering did not work well. ## Open Thoughts Agents - Similar approach applied to agent trajectories and RL environments; also established scaling laws. - Learnings: - Stronger models not necessarily best teachers — he tentatively recalls Qwen models performing better than Claude models ("I think"). - Multiple answers again helped. - **Synthetic rewriting and task augmentation underperformed** expectations. - **SFT contributed most gains; RL was compute-intensive and helped only for "the last few percentages"** — in many enterprise situations SFT works well. ## Enterprise Case Study: Intuit / Credit Karma - Use case: generating explanations for credit card recommendations. - Problem: prompting a model required long compliance rule lists, which **blew up latency**. - Challenge: imbalanced dataset caused post-trained models to **hallucinate numbers** (e.g., 0% APR figures). - Solution: curation recipe adding **tags** alongside prompt-response pairs so the model focuses "on the form rather than the specific numbers." - Outcomes claimed: improved compliance, latency, throughput; model ownership and cost control as frontier models get more expensive. - He notes: "in enterprise settings, we haven't seen lot of successes" in post-training deployments — this is presented as a rare concrete example. ## Tooling & Reference Stack - **Curator**: curates from Hugging Face datasets or collected logs; integrates with Tinker and Fireworks. - Proposed reference stack (which he says "connects back" to a prior talk by Ross on algorithms, environments, compute — noted convergence): - **Environment layer**: building RL environments, measuring quality, version tracking - **Infrastructure layer**: sandboxes, rollout generation, checkpointing, snapshot/rollback for long-horizon rollouts - **Training layer**: SFT, RL - **Prompt optimization**: mentions **Jeppa** (as spoken) — LLM-based reflective prompt optimization for system prompts and harnesses ## Unresolved / Qualified Points - Teacher-model comparisons are hedged ("I think... Claude models, I think") — stated tentatively. - Explanations for multi-answer sampling effectiveness are speculative ("I think the reasoning is probably..."). - No specific benchmark numbers or dataset sizes were stated in the transcript. # Speaker Attribution — Video 01a15ca8-08b7-7a10-adcb-94d5cbf84316 ## Speakers - **Mahesh Sathiamoorthy** (transcribed as "Satyamoorthy") - **Role:** Presenter / sole speaker (guest talk, not the video channel's creator content) - Self-identified: co-founder and CEO at **Bespoke Labs**; previously researcher/engineer at **Google DeepMind** - Delivers entire talk; no other speaker has lines (audience applause only) ## Referenced Third Parties (mentioned, not speaking) - **Nick** — presenter of a previous talk on similar RL-environment work - **Ross** — gave a talk on long-horizon agents; referenced twice (long horizon, and "tags" with respect to hallucinated numbers) - **Alex** — co-founder at Bespoke, chief scientist, professor at UC Berkeley (shown in a tweet) - **John Schulman** — quoted via tweet as using Open Thoughts internally at Thinking Machines - **Microsoft CSO** — referenced via tweet about Open Thoughts - **Intuit / Credit Karma** — enterprise engagement example ## Referenced Sources & Projects (per speaker) - **Bespoke Curator** — synthetic data curation tool - **Bespoke Stratos → Open Thoughts** — reasoning dataset; published as a paper at **ICLR (this year)**; consortium with Stanford, UC Berkeley, UDub - **Open Thoughts Agents** — agent data/RL environment curation - **Terminal Bench** — Bespoke core contributors - Benchmarks: MMLU-style knowledge benchmark (unspecified), **SweepBench**, **TerminalBench**, **LiveCodeBench**, "Amy" (likely AIMATH; ambiguous in transcript) - Teacher models referenced: **DeepSeek**, **Qwen ("Gwen"/"Quinn" transcription variants)**, **Gemini**, **Claude** - Infrastructure providers: **Fireworks**, **Tinker**, **Slime World** (likely "SLM/Scale-world"; ambiguous) - **Jeppa** (likely GEPA) — prompt optimization method ## Promotional Segments - **Company introduction** — Bespoke Labs mission and services (data, RL environments, enterprise post-training) at talk open - **Product mentions** — Curator tool with Tinker/Fireworks integrations; enterprise case study (Intuit/Credit Karma) presented as a concrete deployment success ## Availability Limits - **Comments:** not indicated (no curated metadata supplied) - **Q&A:** speaker states questions will be taken "offline"; no Q&A content in transcript - Slide-only references (paper figures, tweets) are described but not shown in transcript Hey everyone. Um Today I'll be talking about data and uh environment curation for post-training LLMs. And I am Mahesh Satyamoorthy. Um I'm co-founder and CEO at Bespoke Labs. And previously I was a researcher and uh engineer at Google DeepMind. So very briefly I will tell you a little bit about Bespoke and after that the talk will be mostly around open-source work we have done. So Bespoke is an applied data research lab with a mission to help enterprises and frontier labs access high-quality data and RL environments for their post-training needs. So very briefly what we do and what we have done is that last year we put out something called Curator, which is a tool for curating uh synthetic data for post-training with basically SFT. And right after that actually DeepSeek landed and we started an effort to curate reasoning data and that's how we started something called Bespoke Stratos, which eventually formed uh into the project called Open Thoughts, which some of you hopefully know about. And we have also been core contributors to Terminal Bench. Um you know, these days we we do a lot of research and build and ship RL environments. So I was actually looking forward to the previous talk uh from Nick, who's also you know, uh doing something similar. Um and and the other thing we do is we do a lot of post-training and help enterprises uh to get their own custom models, right? That's the name of That's how we ended up with the Bespoke uh title for the company. Um the the other thing I want to kind of mention is there's there is this, you know, you know, in in our industry there are a lot of people who create data, uh, create RL environments, and then there are the, uh, researchers who consume this. But I feel like there is this slight mismatch and it's kind of beneficial for someone to kind of go do both at the same time. And in fact, uh, as you're curating data, you want to put yourself in the shoes of the researcher to see what what does it take to, you know, uh, actually move the metrics on the models. So, that's one of the motivations of how we kind of think about. The other thing I want to kind of talk about is, you know, how, you know, uh, AI has evolved, right? So, early on we used to think about and evaluate models on what they know. Um, for example, this is a this was a very popular benchmark on, uh, testing LLMs on various kinds of STEM, humanities, and all that knowledge. And these days we have all these benchmarks that test, uh, how how how agents are able to do things. We have moved on from knowing to doing, right? So, that's the idea of agents, obviously, and the one of the key principles or one of the key things about agents is that they are autonomous. And there are as I was saying, there are many benchmarks including, uh, SweepBench, TerminalBench, and so on. But ultimately, for many people, what they care about is are these agents autonomous for long durations of time? Uh, Nick had uh, sorry, uh, Ross had a great talk on long horizon, right? So, that's the goal is eventually we make these agents autonomous for maybe few hours or a, you know, few days or, uh, few weeks. And what is it that's blocking the, uh, autonomy of agents? It's basically reliability, right? So, at some point something falls apart. Like either they call the wrong tool or they made a mistake in you know what what not, right? And what what's one lever to improve reliability? There there are of course many. Obviously, you can prompt your way to improving the agents reliability or you can update the harness, you know, the tools and what not. But post training is a very powerful tool to improve reliability or or maybe even pre-training good models, right? So, if you think of Frontier Labs, this is one of their primary mechanisms of improving agents over to to get better \[clears throat\] capabilities in uh various domains or you know uh or for better benchmark numbers or better autonomy for long longer and longer durations. And for post training, one of the popular techniques, as you know, is reinforcement learning. And that's kind of something, you know, a lot of you are excited about is the you know, notion of RL environments. But ultimately for post training, be it SFT or or reinforcement learning, data is the bottleneck, right? So, when I talk about data, RL ends are also something I'm calling it as data. It's just the data is now in a very different shape. Uh again, here, you know, compute is kind of well defined. Models, you know, good sort of models exist and the infrastructure to post train, for example, there are various providers like Fireworks, Tinker or Slime World and what not. So, all of those are somewhat well defined. Most of the places where people struggle, especially enterprises, is that they don't have access to good quality data and RL ends. And this obviously also applies to frontier labs where they have all this infra setup and they are, you know, needing good quality LMs, right? Uh beyond So, that that's one of the This is kind of how we are thinking about it, why to invest time in, you know, doing data research and LMs research. And as a side note, one of the um other there are many other benefits of post training. For example, you can reduce latency or improve cost, throughput, and whatnot. And I will give one concrete example of a post training work we did uh with one of the enterprises. So, in this talk, I will mostly uh cover some of the work we have done in the open source uh community. So, we did some work on curating reasoning data for reasoning models and for uh you know, curating trajectories and uh en- environments for agents. And recently we had an engagement with post training which uh I'll I'll very briefly talk about and some tools on data curation. So, Open Thoughts uh is uh reasoning data set as well as a paper, right? So, we we started this effort last year. As I was saying, this uh we we after Deep Sea came out, we realized that there is uh lack of very high-quality reasoning data in the uh community. Obviously, the labs have access to good data, but outside we didn't have access to data, right? So, we we at Bespoke started this effort called Bespoke Stratus and then we realized that this is actually quite useful. So, we joined um together with various folks in uh Stanford, UC Berkeley, UDub, and so on to create this consortium called Open Thoughts. And we did a lot of work on basically identifying the curation recipe. And we also published this as a paper in ICLR of this year. And this is the main figure of the paper. So, what it shows is like we we figured out a curation recipe and it shows the scaling law, right? So, again, this is last year when Amy and and uh live code bench and these these were some of the popular benchmarks. What we showed is that with this recipe, if you keep, you know, scaling up the data set size, the the, you know, the the it's a scalable recipe, right? The the metrics also improve. Uh it's actually very widely used as well. For example, this is Microsoft CSO tweeting about the work and this Alex is my co-founder. He's a chief scientist and also a professor at UC Berkeley. And this is John Schulman talking about open thoughts that he has he and his colleagues have been using it internally at Thinking Machines, right? And some of their blog posts also reference this. So, I'll be uh talk about how we did the curation for open thoughts. Um Th- This is the pipeline that we used. So, you start with curate you start with a bunch of source questions, right? So, there are various data sets out there that have the prompt response and we choose with the prompts. So, we start with the prompts. These are various sources we have. And then uh if you look at the paper, so if you look at this graph, for any given data point, say if there are 10,000 samples that you want, the question is then how do you choose the questions from all these different data set so that you have 10,000 for the data point. So, the the then there is the aspect around how do you mix these questions? So, uh you can use various methods. So, the paper talks about, for example, using LLMs to check for whether this is a good good question, hardness of a question, and so on. And then you want to filter questions and generate the answers. Again, this is all like driven by LLMs, right? So, this is the curation recipe we did for creating this reasoning data set. And the answer generation is using teacher models, so you can take other reasoning data reasoning models such as DeepSeek or Gwen base models or even Gemini and whatnot. And then you can also filter the answers once you have the answers for these questions. And then you can also, you know, given a question, generate multiple answers or a single answer. So, these are various knobs in the curation recipe. And the systematic way of doing this is like you run ablations and figure out which, you know, in each of these stages what works and you kind of proceed to the next. So, after doing all of this, you get the final recipe, right? So, this you can read this paper. It has lots and lots of, you know, information about how we did the curation. But here are some of the learnings that, you know, some of them are quite counterintuitive. And some of this was also covered in last year's AIE AI engineer conference. For example, sampling multiple answers per question works pretty well. This is something that we it's it's kind of counterintuitive. So, as an example, something else we could have done is we could have had more much many more questions and then just answered them exactly once versus taking one question and answering them 16 times. The I think the the reasoning is probably that it gives like a variety of how reasoning is done, so the the during fine-tuning we also use the the reasoning traces, right? So, I think the diversity helps there. And the other thing we saw is like the stronger teachers are not always the best stronger models are not always the better teachers. And there were a few other counterintuitive aspects around like, you know, synthetic question generation or question answering working whereas answer answer filtering and other aspects not working very well. And after the open thoughts work, which was around data curation for reasoning models such as, you know, deep sea kind of models, we moved on to open thoughts agents, which is very similar, but how do you curate these the data and RL environments for training agents now, right? Not reasoning models. We We have a very similar figure here. Again, we want to establish scaling laws. So, as you increase the data set size, we want to make sure that the curation recipe actually works. Um And again, I'm I'm not going to go into details here, but very similarly, there are various ways of choosing different sources, for example, Stack Exchange and and whatnot. How do you mix the task? How do you filter? Generating the rollouts, choosing the teacher, and so on. And again, these are some of the lessons learnings. Um As an example, even here we saw that stronger models are not necessarily the best teachers, right? So, we found out some some some of the I think uh Um Quinn models were better than, for example, um um Um I'm uh Claude models, I think. And sampling multiple answers again helped in this case. Synthetic rewriting and task augmentation um is something we thought it'll work, but it didn't very work work very well. And the other thing is like in in this whole process of building this open thoughts agent, SFT still contributed a lot to the gains. Um RL was kind of, you know, it's very compute-intensive and for for the last few few percentages, it really helped. Uh but but, you know, in many of the situations, for example, in enterprises, SFT actually works pretty well, right? And here is one concrete example I wanted to share on um uh actually deploying something to production, right? By post-training. So, we have seen lot of people talk about post-training, but in enterprise settings, we haven't seen lot of successes, at least I haven't seen uh that. Here is a very concrete example of uh with Intuit, there is a this app called Credit Karma, which if you install, there is a page place where you can uh the the app gives you a reasoning as to why a credit card has been recommended. And this you can prompt a model to do this, but one of the reasons one of the places where it fails is that the you know, it it it's not always compliant. So, you have to have a long list of rules to make sure the responses are compliant. And that actually blows up the latency. So, answer here is like you want to curate data and post-train, right? Seems kind of straightforward, but one of the things that we ran into is um the data set can be quite imba- imbalanced and lo- lots and lots of places, for example, you will have 0% APR and the model after fine-tuning can kind of hallucinate the these uh numbers. So, this again kind of ties back to what Ross talked about sometime back with the respect to the tags. And we kind uh we we created this specific uh curation recipe where instead of just having these uh question the the prompt response pairs in plain language, we added these uh tags, which helped the model to focus on, you know, uh uh, the the kind of form rather than the specific numbers itself. And that gave a big boost. And, uh, we we saw that, uh, the the overall the complaints metrics improved, the latency improved, the throughput improved. And eventually, you know, they are able to own the model, right? As frontier models improve, they don't need to kind of go and, uh, uh, update it. And also, as we see now, the, uh, frontier \[clears throat\] models are also getting more and more expensive. And, you know, this kind of gives them a very good way for owning the model and also, uh, lowering the costs. I think with that, I want to briefly touch upon, um, a you know, Curator, the tooling that we had built last year, um, which is for curating reasoning data. So, uh, what it does is you can basically, uh, you know, um, specify the you can either go with, say, a hugging face data set where you have various prompts. Or, uh, in many situations, you may have collected logs and you want to, uh, get the responses and fine-tune a model. So, this Curator kind of makes it pretty easy to do that. And it comes with, uh, integration with, you know, uh, Tinker and Fireworks. And this this is again the tool that we used, um, originally for curating open thoughts. And here is a very very detailed diagram of what we are building today. But, uh, this again connects back to, uh, what Ross was talking about, where he was talking about algorithms, uh, environments, and compute, right? So, it it feels like, you know, we are kind of converging on something very similar. So, if you think about, uh, the the stack that is needed to save not just curate these RL environments but to post chain models. One of the things you need is obviously handle on like how do you build these RL environments? How do you measure the quality? How do you track the different versions and so on? So that's one of the layers. And below that you want various infrastructure to um sandbox you sandboxes, right? So spin up the rollouts to to spin up the sandboxes to generate rollouts. And especially if you have long horizon rollouts, then maybe at some points you need to do a checkpointing and then you need to be able to snapshot or rollback to something else, right? So that's the other the lower level you know compute and orchestration. And at the top I have been giving examples on post training. So there is all this layer around like how do you do SFT? How do you do RL and so on? But there is also this method called Jeppa which is around which is on prompt optimization. I don't know if you if you guys have heard of it, but you can use LLMs itself to uh to to kind of optimize the prompts based on reflection. So that also works pretty well for updating the system prompts and also the harnesses. So this is kind of I feel like you know the the new architecture or the new reference stack for how how at least we are building and how many others are building the the stack on how to build the RLMs and then also post chain agents. I think with that I'll end the talk and you know happy to take questions offline. \>\> \[applause\]