Talk by Kobie Crawdord (Snorkel) on task fidelity scaling laws for fine-tuning: using TerminalBench-style agentic tasks, an experiment held model, compute, and task count constant and found low-quality tasks improved the base model by ~1% versus ~6% for high-quality tasks (a 5x difference in training uplift from task quality alone). The talk analyzes what separates the two: accepted tasks averaged more tool calls, lower pass rates, and more output tokens (genuinely harder problems), and their failure modes were cleaner, whereas rejected tasks failed due to mismatches between requests and tests or missing context for implicit dependencies — concluding that ambiguous specs produce noise rather than difficulty. # Channel Identity Observations — "Task Fidelity Scaling Laws" (Snorkel) ## Explicit Self-Description (Creator Claims) - **Speaker role and employer**: Speaker identifies as **Kobie Crawford, developer advocate at Snorkel** — "My name is Kobe Crawford. I'm a developer advocate at Snorkel. We are the frontier AI data lab." - **Company scope claim**: Snorkel "produce[s] data sets for foundation models to help climb on" and claims Stanford AI research lab origins ("This company's origins actually begin from a Stanford University AI research lab… part of… the CEO's PhD thesis"). - **Core company thesis stated**: "the core thesis has been that the quality of data is critical… since they got started as a company in 2019." - **Claimed methodology**: "we're using human expertise and having experts in the loop for generating the data, and we have a strong feeling that the expert in the loop is an important element of delivering data quality" and "we can deliver quality at scale." - **Claimed products/programs**: "we have an open benchmark grants program… partnering with folks who are developing benchmarks and evaluations," and a "leaderboard… a couple of benchmarks that Snorkel built and curates ourselves… one called [A]gentic [C]oding which is focused specifically on these terminal bench style tasks." ## Observable Presentation (Demonstrated Patterns) - **Format**: Conference/talk presentation with live audience Q&A ("Thanks very much and then I'll take the next question"), ending with follow-up questions and time pressure ("do I still have 1 minute left?"). - **Content scope demonstrated**: Technical research presentation on **agentic task quality vs. data quality for RL training**, including empirical results: "about a 1% improvement with using the low-quality tasks… about a 6% improvement with the higher-quality tasks," "a 5x uplift difference based on just quality." - **Intended audience (demonstrated)**: ML/AI practitioners and researchers familiar with agentic evaluation tooling — references to "Harbor framework," "Open Env," "terminal bench," "GPT 5.x," "Codex," "sign at 4.5," RL training runs, and inter-annotator agreement are used without basic explanation. Audience members ask specialist questions (benchmark contamination effects, task underspecification, multi-step verification). - **Evidence rigor demonstrated**: Four-part task acceptance criteria (achievable, non-trivial, functionally correct, environment reliable) with accepted/rejected buckets; failure-mode categorization; controlled RL comparison "with the same model the same compute budget the same number of tasks." - **Positioning behavior**: Repeatedly ties results back to company products and claims — "the way that Snorkel generates data sets… we can talk more about how our platform… we scale and we can deliver quality at scale" — and closes with links to a "research page" and leaderboards. ## Audience & Scope Summary - **Audience**: Technical AI/ML professionals (researchers, ML engineers, benchmark developers) at an industry conference. - **Claimed scope**: Snorkel as a "frontier AI data lab" delivering expert-in-the-loop, high-quality training/evaluation data and benchmarks at scale. - **Demonstrated scope**: One research talk validating a single thesis (data/task quality drives RL training outcomes) with empirical evidence and product-oriented framing. # Delivery Observations — "Task Fidelity Scaling Laws" (Kobie Crawford, Snorkel) ## Tone & Register - **Conversational, discussion-style delivery**: frequent filler and self-restart patterns throughout. > "uh uh as a company in 2019 is that the the core thesis has been" > "that that what what we do in the Agentic context" - **Confident but hedged when uncertain**: openly admits gaps in analysis rather than bluffing. > "Um a good question. Um the interesting thing about so I don't have a specific quick direct answer" ## Recurring Phrases & Verbal Habits - **Repetition for emphasis** is a dominant pattern — repeated words and phrases both within and across sentences: > "the the the the the tests that we use" > "that that that's the case" > "on on on on just quality" - **"The the the X is that..."** framing to introduce claims: > "the core thesis has been that the quality of data is critical" - **Signposting transitions**: uses "So," and "And so," repeatedly to link sections: > "So, we're working with uh a flow..." > "So with that in mind we then take it as a..." - **"kind of" / "sort of"** as softeners, especially in Q&A: > "it's just like a failure that's not super meaningful in terms of like it's just a tactical thing" > "something where it's just like sort of like whether or not the models are actually improving" ## Teaching Habits - **Level-setting before claims**: explicitly frames scope before presenting results. > "So let's look at those comparisons for just a moment and make sure that when we actually take a look at it and level set" - **Empirical rigor framing**: repeatedly distinguishes belief from evidence. > "we actually want to verify that that that's the case" > "We actually want to dig into the failure modes a little bit as well" - **Numeric anchoring**: key results delivered with concrete figures. > "about a 1% improvement with using the low-quality tasks... about a 6% improvement with the higher-quality tasks" - **Definition-by-list**: breaks concepts into enumerated criteria. > "the task is achievable, that is non-trivial, uh that is functionally correct" ## Q&A Communication Patterns - **Aggressive agreement as a bridge**: responds to pushback with affirmation before pivoting. > "Certainly, certainly. Uh 100% agreed" > "Yes, absolutely. In fact..." - **Defers when memory fails**, without fabricating: > "trying to remember the name of it, but it was about sort of like..." - **Time-awareness in the final answer**: negotiates remaining time on-mic. > "Um do I still have 1 minute left? You only have 1 Okay, brilliant." ## Stability vs. Isolated Moments - **Stable patterns**: filler/repetition, "So"-based transitions, rigor framing, and numeric anchoring recur throughout. - **Isolated**: the capitalized transcript ending ("THE NEXT QUESTION. CAN I TAKE ANY FOLLOW QUESTIONS?") appears to be a transcription artifact of projection volume, not a stable delivery feature — **uncertain**, based on one occurrence. - The "expert in the loop" theme appears mainly in the closing pitch section and may reflect talking points rather than a consistent through line — **limited repetition**. # Observation Record: Task Fidelity Scaling Laws — Kobie Crawford, Snorkel ## Speaker Identification - **Kobie Crawford** (transcript renders name as "Kobe Crawford"), Developer Advocate at Snorkel — sole identified presenter. - **Audience members** (unnamed) posed follow-up questions during Q&A. ## Company Context (as stated) - Snorkel describes itself as **the frontier AI data lab**, producing datasets for foundation models to "hill climb on." - Origins: Stanford University AI research lab; early work tied to the CEO's PhD thesis, became an open-source library; company founded **2019**. - Core company thesis: **data quality is critical** across all cases. ## Core Research Claim - **Task quality and data quality are "largely the same thing"**; the data-quality premise governs agentic tasks as well. - Research objective: empirically verify that task quality affects training outcomes, not merely assert it. ### Scope and Qualifications - Context specifically: **agentic terminal-bench style tasks** — containerized environment plus task definition. - Speaker acknowledges architecture and harness changes are "obviously impactful," but claims data quality underlies all of it. ## Task Quality Definition Four criteria stated by Crawford: 1. Task is **achievable** 2. Task is **non-trivial** 3. Task is **functionally correct** (logic plays as expected) 4. **Environment is reliable** Process: tasks passing all four tests → **accepted bucket**; otherwise **rejected bucket**. These buckets define high- vs. low-quality task comparison. ## Comparative Analysis (Accepted vs. Rejected Tasks) Tools used: **Claude Sonnet 4.5** ("sign at 4.5" in transcript) and **Codex** (GPT 5.2, 5.1, sometimes 4.0). Findings for accepted tasks: - **~2x more tool calls** — more difficulty, more steps, more external tool engagement - **Lower pass rate** — higher intrinsic difficulty - **More output tokens** — more reasoning ### Failure-Mode Caveat (stated by speaker) - Lower pass rate might reflect **failure modes without meaningful signal** — so failures were categorized: - Meaningful failures (model not reaching a logical conclusion) - Degenerate/environmental failures (no model could complete the task) Findings: **logic error** and **incomplete task** categories showed a **reversal** of relative percentages between accepted and rejected tasks. General takeaway: **accepted tasks produce "cleaner failures"** — genuinely harder tasks useful for hill climbing, vs. tactical/environmental noise. ## RL Training Result - Setup: same model, same compute budget, same number of tasks in each arm. - Result as stated: - Low-quality tasks: ~**1% improvement** over base model - High-quality tasks: ~**6% improvement** - Characterized as a **"5x uplift"** attributable to quality alone ## Methodology Claims - Snorkel uses **human experts in the loop** for dataset/RL environment generation; Crawford states the team has "a strong feeling" the expert-in-the-loop element is important to quality. - Emphasis: quality should be the first consideration for data. ## Benchmarks Mentioned - Snorkel builds and curates its own benchmarks, including one called **"agentic coding"** (transcript: "a genetic coding") focused on terminal-bench style tasks, with claimed additional rigor. ## Q&A Positions ### Q: Does a mix of accepted/rejected tasks pull the model down? - Crawford: **no direct answer from this analysis**. But in benchmark comparison work (terminal bench 1 vs. 2 variants), tasks that "never get completed" because they *can't* be completed became a **source of noise masking whether model improvement was actually happening**. ### Q: Does input prescriptiveness affect difficulty? - Crawford: **yes** — under-specified tasks are a common rejection cause. Two cited mechanisms: - Tests on the back end expect things **never requested** in the task definition (mismatch). - **Implicit dependencies** required by tests but not fed into the model's context. - Audience follow-up: classifying incomplete tasks as failures may be an issue, since real problem-solving is iterative, not one-shot. - Crawford: **"100% agreed"** on that framing generally, but benchmarking tasks are built to be verifiable on the back end; in principle the learned skills should still transfer to unverifiable/iterative contexts. ### Q: Future work on non-verifiable, long-horizon tasks? - Yes — multiple projects. Coding and math make verification easy; "fuzzy" domains are different. Snorkel runs an **open benchmark grants program** partnering with teams on less-verifiable evaluations, including one (name not recalled) on emotion-level, human-centric tasks where **multiple outcomes are possible and scored along a spectrum**. ### Q: Inter-annotator agreement and multi-step task complexity? - Crawford: platform combines **human annotators and LLM judges**, using **rubrics** with longer lists of criteria. Goal: high inter-annotator agreement among humans and between LLM judges and humans. Acknowledges explicitly verifiable test-pass/fail domains (like this one) are easier than others, but the same guiding principle is applied across domains. ## Unresolved Tensions (preserved as expressed) - Speaker's own caveat that lower pass rates could stem from meaningless failure modes — addressed via categorization, but percentages were described qualitatively ("you can see a reversal") without specific figures quoted in the transcript. - No direct answer available on the accepted/rejected mixture question. - Acknowledged tension between verifiable benchmark tasks and real-world iterative problem-solving (audience-raised; speaker agreed but defended the verifiable approach). # Speaker & Provenance Observation Record ## Video - **Title:** Task Fidelity Scaling Laws — Kobie Crawdord, Snorkel - **Video ID:** `01a15ca9-156b-7749-8b9a-6c1ee93e37fb` ## Speakers & Roles ### Primary Speaker: Kobie Crawford *(transcript spelling varies: "Kobe Crawdord"/"Kobe Crawford"; title lists "Kobie Crawdord")* - **Role:** Creator/presenter — self-identified as **Developer Advocate at Snorkel**, described as "the frontier AI data lab." - Delivers the main talk on task quality vs. data quality in agentic RL training. ### Audience Questioners (unidentified) - Multiple unnamed Q&A participants asking follow-up questions; no names or affiliations given in transcript. - One questioner offered a follow-up comment about tasks not needing one-shot completion ("classifying them as a fail could be an issue"). ### Session moderator (implied) - Speaker managing time ("You only have 1 [minute]... This is the last question") — identity not stated. ## Provenance & Sources Cited by Speaker - **Snorkel origins:** Stanford University AI research lab; CEO's PhD thesis; open-source library precursor; company founded **2019**. - **Models used in experiments:** "sign at 4.5" *(unclear transcription; possibly Claude Sonnet 4.5)* and **Codex** with "GPT 52, 51, and sometimes 40" *(unclear transcription; possibly GPT-5.2/5.1/4.0)*. Speaker notes tests ran "some months ago." - **Frameworks referenced:** Harbor framework, Open Env, terminal bench (Terminal Bench 1 and 2). - **Benchmarks:** Snorkel-built "genetic coding" benchmark *(likely "agentic coding" — transcription unclear)* for terminal-bench-style tasks; a Snorkel leaderboard. - **Programs:** Open benchmark grants program partnering with external benchmark developers; unnamed partner organization working on emotion-related/human-centric evaluation. - **Links shared:** Snorkel research page and leaderboard (no URLs in transcript). ## Claims Reported (not evaluated) - Accepted tasks averaged **2x tool calls**, lower pass rate, more output tokens than rejected tasks. - RL training results: ~**1% improvement** (low-quality tasks) vs. ~**6% improvement** (high-quality tasks); ~5x uplift difference. - Task quality defined by four criteria: achievable, non-trivial, functionally correct, environment reliable. ## Promotional Segments ⚠️ - Talk opens with company positioning ("We are the frontier AI data lab"). - Closing segment promotes Snorkel's **research page, leaderboard, benchmarks**, platform capabilities, and **expert-in-the-loop data services** ("we scale and we can deliver quality at scale"). ## Availability & Uncertainty Notes - **Comments:** No metadata supplied; comment availability unknown. - **Transcription quality issues:** Speaker name spelled inconsistently; several model/benchmark names garbled (flagged above with uncertainty). - Q&A portion is partially attributed; questioner speech embedded without speaker labels. My name is Kobe Crawford. I'm a developer advocate at Snorkel. We are the frontier AI data lab. And what that means is that we produce data sets for foundation models to help climb on. So our research team is highly integrated with the work that we do in terms of our production work. And we put a lot of emphasis on how we integrate research in that. This company's origins actually begin from a Stanford University AI research lab and uh the work that they were doing there actually was part of one of the the CEO's CC PhD thesis and then that became a library that was used open source for a while and then we've grown into focusing on delivering things uh uh with data sets for our customers. Uh one of the things that's been a consistent through line for Snorkel since we the since they got started in uh uh as a company in 2019 is that the the core thesis has been that the quality of data is critical and that you the data that you're looking at you want to make sure is is top quality and in all those cases. So we look at how that applies to the data sets that we provide as well as as things move into the agentic space uh how that applies to agentic tasks. And what we wanted to show in this context is how data quality impacts uh things in the context of task quality and and the task the task quality and data quality are largely the same thing. So we're going to talk about the particular research objective here was looking at again uh how task quality affects the training outcomes that you get when you're trying to improve models. And uh then from there, we're going to talk about the techniques and and the path that we chose to to to verify that these behaviors were actually happening for us. Uh so, does the task quality actually matter is uh you can you can tell from where we from our thesis about it that we actually think that of course it does. And uh the way that we're going to break down talking about that is just make sure you understand where we're approaching this. We're talking specifically in the context of uh Agentic uh terminal bench style tasks. Uh so, we're working with uh a flow that is going to be a containerized environment and then a task uh definition within that. Um we want to show that when you're looking at how uh you look at the tasks themselves and how they're built that that what what we do in the Agentic context uh is still also governed by the same data quality premise. So, that applies in terms of talking about task quality as well as data quality. And so, if your architecture changes, if the harness that you're using changes, these kind of things, all of those things are also obviously impactful things, but underlying all of that is still that data quality is at the center of it. So, here what we're doing actually puts some specific kinds of rigor and deliver delivering empirical evidence to validate that this is true. That we want to just not just sort of say we accept this as a as like a thing that we like to say is true. We actually want to verify that that that's the case. So, in the definition of task quality, we're talking about basically four core things. Um if you've worked with these kinds of environments, the Harbor framework, uh Open Env, uh when you built tasks for Agentic purposes, what we're talking about in the context of evaluation, benchmarking RL, is that we are creating an environment in which that is going to run. We have it containerized for reproducibility and isolation. And that also allows you to parallelizing for rollouts of these kind of practical elements of how that works. Uh inside of that, you have uh looking at what's in the logic of the task. You want to talk about um that the task is achievable, that is non-trivial, uh that is functionally correct, that the logic actually plays as expected, and then the environment itself is reliable. And so that that environment reliability is a is a key as well. Those four criteria uh for us as we work on this, the Snorkel team has built in our research uh research harnesses, uh we've built a setup where we verify all four of those criteria. And in our tests to verify those criteria are the the the the the tests that we use. If a task passes all of those tests, then we consider it an accepted task is accepted and then becomes something we can use for our training and research and purposes. And then if it's not accepted, then it would be uh put in the rejected bucket. And we use those two buckets as a then basis for talking about how we're going to compare what is a high-quality task, the accepted ones, to what is a low-quality task. So let's look at those comparisons for just a moment and make sure that when we actually take a look at it and level set, does the acceptance criteria that we use tend to correlate with actual performance behaviors that we think we want to see then how how they differ. Uh and so the way that we did that was we used uh sign at 4.5, so it was obviously some months ago, uh and Codex uh which had like uh GPT 52, 51, and sometimes 40 uh included in terms of the tests that were run. But we um used those two uh for uh running these uh tests. And we compared how those tasks uh were completed. Uh and and in the completions, we found that uh our accepted tasks averaged twice as many tool calls, uh, demonstrating more difficulty, more steps needed, and, uh, and more engagement with the external tools. A lower pass rate, so higher difficulty, uh, intrinsically. And then, uh, also more output tokens needed, so there was more reasoning that was done by the models to actually do that. Um, the failure modes, however, So, for let's go back to the pass rate for a second. It's possible in the context of a lower pass rate that you could have failure modes that actually don't show meaningful signals. So, we actually want to dig into the failure modes a little bit as well. And so, the next step here is to just say like, "Okay, what does it mean to be talking about those failures?" Uh, to do that, we broke down the failures into categories, uh, to identify the kinds of failures that represent something that's meaningful, like this model is not performing uh, the task completely because the model's un- not not achieving a logical conclusion that it needs to, uh, versus a failure that's more say of a kind of a degenerate case where you have a problem that is an environmental problem, something that like literally makes it so like no model would be able to solve that problem and not complete that task at that in this particular flow. So, we have a breakdown of those, and then given those again, we wanted to compare the accepted versus rejected tasks and see where the failures are occurring in those tasks and see what we get from that. Um, so, here's a summary of breakdown of of of each of these things across, uh, the percentage of failures that we saw in each of these different categories, and you can see I like to highlight in particular, uh, the logic error and the incomplete task, uh, uh, bars and just observe that, uh, you can see that the, uh, in in each case you can see a reversal of like which one had a higher percentage appear in those there. And then this breakdown, um, even with this, uh, with a with a comparative uh, analysis based on percentages of failures, you can see that you can see the difference in terms of like where you see the the the the un- overrepresentation or underrepresentation of these kinds of failures across the types of failures that we uh categorized and again between where the rejected tasks versus where the accepted tasks had errors. Um the general tendency that we are taking away from this is that the accepted tasks are producing cleaner failures. These are failures due to the task itself being more difficult, truly more difficult that that the steps that it needs to to accomplish are are more difficult and that that means that this is a test that would be actually very useful for the model to be able to hill climb on, provide some data samples that could help it actually be able to be improved in terms of those performance patterns versus something where it's just like a failure that's not super meaningful in terms of like it's just a tactical thing that's happening inside of the context that's not working. Um so with that in mind we then take it as a you know, we've accepted that we've actually put together enough analysis that gives us a pretty strong sense that the accepted tasks are also higher quality tasks in the main and the same thing again that given that we have a differentiation between the higher quality tasks and the lower quality tasks, now we want to actually see can we see an impact on model performance when we use it. So following forward from that we just actually run a a training run, RL training run with the same model the same compute budget the same number of tasks in each case and and then look at the the difference there. So that that's where we've level set that we have a set of test tasks we consider high quality, level set of tasks that we consider lower quality and that that usage is going to help us say something about that. So we trained it twice and we wanted to see what we got. And the performance uplift is uh actually very meaningful. Uh we're talking about about a 1% improvement with using the low-quality tasks. So, after the RL training was done, the low-quality tasks only improved the base model by about 1% improvement, but the improvement was about a 6% improvement with the higher-quality tasks. So, that uplift of the 5x uplift difference based on on on on just quality is really striking from our point of view. We think that that you know, cements the the the intuition that like it really is important for the data quality to be high. Uh the way that Snorkel generates data sets uh and and the way that we put together RL environments, we're using human expertise and and having experts in the loop for generating the data, and we have a strong feeling that the expert in the loop is an important element of delivering data quality. And between those, that ultimately gives you you know, uh we can talk more about how our platform how we use our platform to help make that work by our experts using that we scale and we can deliver quality at scale. But, the but the key is that we have we want to put the emphasis on making sure that quality is the first thing that people think about about what you need your data to uh to what you need from your data to make sure that you're getting good results. So, that's the that's the end of what we got out of that. Hopefully, if you have additional questions, we have just a couple of additional minutes, but um here's a couple of quick links to the research page of the Snorkel summarizing how our research team works and what we do and what we put our emphasis. And then and then the leaderboard is pointing to a couple of benchmarks that Snorkel uh built and curates ourselves of that are similar to things that you've seen. We have one called a genetic coding which is focused specifically on these terminal bench style tasks. And we're doing the same kinds of of evaluation but we won't we put apply a certain kind of rigor to how we're going about it that we want to make sure that people could see the difference. Thanks very much and then I'll take THE NEXT QUESTION. CAN I TAKE ANY FOLLOW QUESTIONS? I'LL START HERE. I'M JUST KIND of wondering if you would look at um another kind of ABC I guess of like all the tasks projected accepted to see if it's like if is there uh if there's like an effect of the rejected tasks pulling it back down or actually the model can get over it as long as all the tasks are in there. Um a good question. Um the interesting thing about so I don't have a specific quick direct answer about like from that analysis that we did in this particular case. Um but um one of the things that we saw in surveying and and working with for example working with the terminal bench team um and looking at the tasks that were in terminal bench one versus what we did for terminal terminal bench two and in some other contexts around some of this we bench um some of the variants of this we bench we've been some doing some analysis internally to compare the various public benchmarks that are out there and looking at what what we see across those things and when we look at that um you certainly see that the uh the the the failure rates and like which you know where the where the model models have been improving over time um whether they're getting to saturation faster or the the benchmarks themselves are getting to saturation faster. Uh sometimes we ended up seeing this sort of from noise about that because you end up with a certain number of tasks that never get completed. So then we start to see that like these tasks will never be completed literally just because they actually can't be, right? Yeah, and then because of because of that we ended up like finding that that was actually sort of a source of noise in the in the process of actually evaluating whether the model improvement was actually happening. So, in that way I can't say it's like sort of like whether or not the models are actually improving ended up being something that ended up being more of a masked by like the quality of task issues as opposed to being something where we could tell did the models actually improve or not despite the their presence. It ended up being some more of a source of noise. Um I saw a couple of other questions right here. I have Do you know Do you have any sense of how the input is making this task You know, there is a correlation sometimes input can be a little bit less prescriptive could make it harder or could be very prescriptive make it easier. I don't know, but I'm just wondering you encountered something like Yes, absolutely. In fact, the way that we've been looking at it a lot of times what makes a task sort of a one of the rejected task is it being under specified in terms of like you know when you when the task is defined in a way that the desired testable outcome is not clearly specified in the test definition up front, but then on the back end the tests themselves expect certain things to pass that were never actually requested. Those kinds of mismatches are so the sort of some of the places where you can see where the task becomes or at least appears to be harder because the tests don't match with the the requested task set up. Also, sometimes there are like implicit dependencies in the in the testing that that the task doesn't specify be there and then without knowing that a dependency is required in the first place and what hasn't been fed into the context of the model, then the model doesn't even have the the right context to be able to to approach those dependencies. Just a follow up or correct thing. \>\> I mean think classifying them as a fail could be an issue because not every task needs to be complete. You know, there's iteration, there's a journey. Typically, when we solve problems, it's never a one-shot kind of thing in most of the problems in the world. So, yeah, uh differentiating that with the underspecified could maybe could I don't know, but \>\> Certainly, certainly. Uh 100% agreed that that those kinds of things about like what we ultimately want the models to do tends to go like that. In the context of building benchmarking tasks, we work to make it so that we have something that's verifiable on the back end. And that that that in principle, if we do it right, that the skills that are being learned are still going to be more applicable in the context of unverifiable results or things where there's going to be an iteration that needs to occur following the the step that you're working on currently. Yeah, yeah, so. Thank you. Sure, sure. Yes. Um in terms of like future challenges, next steps, like are you working on tasks that are not as straightforward verifiable and then maybe more towards the very long horizon? We certainly are looking at all of those. Uh and different projects of of ours are working those different spaces, especially once you get outside of the places where verification isn't easy and coding and and math make it straightforward and then things that are more fuzzy are are different. We're uh we have an open benchmark grants program that we're working with where we're partnering with folks who are developing uh benchmarks and evaluations in in some more of the less verifiable areas. And uh there's a lot of very interesting stuff that we're doing that has to do with uh one of them that uh there's a a a a an interesting organization uh that's working on um trying to remember the name of it, but it was about sort of like uh really looking at things that that involve like sort of emotional like level things and stuff like that. It's very lots of very human-centric uh thinking. And so there even even the notion of like, you know, what is correct or not is something where we want to have like actually sort of multiple possible outcomes and then score them differently. But, you know, but have them all sort of fit on the spectrum somewhere. And so there's there's a lot that we're trying to do in the in the the various things in a lot of different dimensions. Um but yeah, so it's very interesting space. One last question in the back. Yeah, I was I was just going to bring up the discussion of this conversation. Mhm. How does your system handle this and the inter-annotator I guess Oh. I guess in the in the multi-step task Yeah. right now it's maybe more reliable like you said the answer is a bit discretionary but like according to that but in the world of annotators there's many possible possibilities I'll speak to inter-annotator inter-annotator agreement sort of centrally and then the complexity you're you're you're talking about 100% is something that like the longer the horizon, the multiple steps involved in the different dimensions are are an issue. This is the last question, yeah? Or we are done with time? Um do I still have 1 minute left? You only have 1 Okay, brilliant. Um so the way that our platform works we actually do a number of things to bring together human annotators as well as using LLM judges. Um and that's partly to help us replicate and scale what our human annotators are are delivering, but also these these kinds of agreement um we feel like the way that we are doing things with rubrics these days and providing sort of like a longer list of of of data points and and and criteria that need to be met that as we sort of build out of set of rubrics that then can be used both by LLM judges and people that we're actually sort of like looking at high-level qualitative things as well as individual sort of like more quantitative comparisons. And so through a long through a longer list of things it builds on a rubric and then using the human annotators and the experts to help us give us that the information the ground truth information that we can inform LLM judges to to look at. We're actually looking to make sure for example that we actually test and get inter inter annotator inter annotator agreement very high between both individual humans as well as between the LLM judges and humans and then use all of those comparisons to to to do quality assessment. So it's part of our assessment process and so we and we use that so for for each of these kinds of of of tests. So in the context here where it's explicitly verifiable tests will pass or tests will fail it's still obviously an easier domain than others but we still keep using that sort of guiding principle apply across all these domains. All right, well thank you very much. Really appreciate your time. It's great to have you.