Talk by Sean Cai, "State of Data," arguing that the scarce AI asset is shifting from static datasets to process data — reasoning traces, decision sequences, failures, and verified outcomes that capture how real work is done. Topics include types of data, compute/data/talent tradeoffs, verifiability axes, benchmark limitations (illustrated with finance benchmark tests on GPT 5.5 and Opus 4.8), robotics as a counterexample, and why durable value lies in live data pipelines plus retraining infrastructure rather than leaderboard climbing. # Observation Document: "State of Data — Sean Cai, Independent / State of Data" ## Video Identity Signals ### Presenter Positioning - Speaker presents as an **independent analyst/commentator**, explicitly disclaiming corporate representation: > "I'm here to not deliver an agenda on any company's behalf but just to expose a lot of alpha in data markets." - Self-identifies as a **writer with an existing body of published work**: > "I have this piece called the TAM is in a vertical it's all of labor." > "all of which are online" > "I put together this very rudimentary sort of chart online in one of my pieces" - Refers to original concepts developed in prior writing: **"Antikythera mechanisms," "type one versus type two data"** ("You'll hear me come back to this mechanism I call the Antikythera mechanisms"). - Claims original proprietary tooling: > "I basically have an internal version of Val's AI and a lot of private benchmarks." ### New Venture Announcement (Explicit Self-Description) - The speaker uses the talk to **publicly announce a new venture**: > "I'm working on something new. This is the first public announcement of it. I'm building a ticket through mechanisms. I'm working with a lot of real-world companies to monetize their data assets, but moreover, help implement RL as a service." - Explicit audience-acquisition intent: > "If you guys want to talk about it afterwards, I'm on Twitter. I always write a lot on Twitter and Substack." ## Intended Audience - **AI conference attendees** (contextual: "this is probably going to be a little different from what you've seen so far at AI conference"). - Sub-audiences addressed directly in the talk: - **Researchers**: "for the more researcher types in the audience"; closing advice "if you're a researcher, stop outsourcing your definition of realism to the same vendors you buy your eval's and tasks from." - **Builders/founders**: "if you're a builder, your moat's not the data, it's the sort of pipeline into real-world work." - **Robotics researchers** (acknowledged in crowd): "Some of you guys are probably robotics researchers in a crowd." - **SF tech industry insiders**: "maybe some of you guys here in SF have heard this a little too much." ## Claimed Scope of Content - **Data markets analysis**: unbundling of vertically integrated vendors ("Surge or Scale AI"), vendor fragmentation, "fog of war" between labs, vendors, enterprises. - **Proprietary frameworks presented**: three-axis verifiability model (asymmetry, veracity, proliferation), Type 1 vs Type 2 data, model improvement equation (compute/data/talent). - **Benchmarking critique**: claims most public benchmarks are "quietly fake," illustrated with private eval runs of finance tasks (ARR waterfall, LBO memo, long-short pair trade). - **Market prediction methodology**: using data-market spending as a leading indicator ("Anthropic was spending a lot on cybersecurity data... what do you know happened like 2 to 3 months after?"). ## Observable Presentation Patterns (vs. Creator Claims) - **Claim**: independent, no company agenda. **Demonstrated**: talk concludes with promoting a personal venture — the "independent" framing coexists with commercial self-promotion. - **Claim**: insider knowledge of opaque industry ("nobody seems to know how Remarque handshake and a lot of these folks actually produce data"). **Demonstrated**: anecdote "I'm recording a handshake. People don't notice an incredibly large amount of their revenues are enterprise now" — asserting private information without named sources. - **Style demonstrated**: concept-heavy, framework-driven talk with named coined terms, direct audience address, humor ("taking a Swiss Army knife and using the screwdriver bit to cut cheese"), and audience laughter/applause indicating a live conference setting. ## Metadata Note - Curated metadata was **not present** beyond the title; the title itself ("Independent / State of Data") corroborates the independent-analyst framing. # Delivery Observations — "State of Data," Sean Cai ## Tone & Overall Register - **Conversational-insider tone**: Speaks like a peer briefing an informed audience, marked by hedged asides: *"you know," "sort of," "I would say," "I think."* - **Anti-corporate framing upfront**: Explicitly distances from vendor agendas — *"I'm here to not deliver an agenda on any company's behalf but just to expose a lot of alpha in data markets."* - **Self-aware candid moments**: Leans into disclosed indiscretion as a rhetorical device — *"I maybe shouldn't say this in a public setting."* ## Vocabulary & Coinage The speaker repeatedly **invents or repurposes named concepts** as teaching anchors: - *"I call the Antikythera mechanisms, which is this sort of bespoke systems that translate messy business context into Evals"* (revisited later: *"the Antikythera mechanisms that I talked about before"*) - *"type one versus type two data, uh contrived versus non-contrived"* — then explicitly flags it as his lexicon: *"This is the vocabulary. I'll use all talk."* - *"a sort of fog of war"* for market opacity; *"benchmark psychosis"* for noisy evaluation ## Recurring Teaching Habits ### 1. Historical/Industrial Analogies A consistent explanatory move — mapping AI markets onto past industrial shifts: > *"Data is to the white collar revolution what coal and iron basically is to the Victorian age"* > *"Railroads built company towns. They charged tyrannical rents. They got nationalized as soon as the automobile moved in."* Also: *"It's basically like taking a Swiss Army knife and using the screwdriver bit to cut cheese and like concluding the knife is broken."* ### 2. Numbered/Triplicate Structuring Favors **three-part taxonomies**: - *"compute, data, and talent"* - *"asymmetry of verification… Veracity of verification… proliferation of verification"* - Ends with enumerated infrastructure jobs: *"There's like five jobs…"* (lists at least three) ### 3. Sequential-Trace Prediction Framing Presents market maturation as an ordered chain, twice: > *"after code, we went to search, and after search, we went to finance, and after finance, we went to healthcare and law"* ### 4. Referral to His Own Writing Repeatedly points to his published work as an extension of the talk: > *"all of which are online… in case any of the particular topics I talk about are are particularly interesting and you want to dive in more"* > *"I always write a lot on Twitter and Substack"* ### 5. Signposting & Time Management - Explicit pivots: *"So, look, you know, going back to the start"*; *"Let me bring it all together"*; *"I'll close on this."* - Mid-talk compression: *"I'll actually just skip to the more interesting part."* ## Audience Address & Interaction - Direct second-person appeals: *"maybe you've been approached by Mercari to buy your Slack logs"* - Audience acknowledgment with humor: *"Some of you guys are probably robotics researchers in a crowd"* — followed by a joke about *"100,000 hours of like iPhone video from my friends in India"* (audience laughter). - Repeated offer of post-talk availability: *"happy to talk more about like emerging infrastructure needs afterwards if you want to."* ## Rhetorical Disclosure Patterns - Frames claims as insider knowledge: *"tell you what's really going on behind the scenes in a in a very murky landscape"* - Cites private tooling as evidence authority: *"I basically have an internal version of Val's AI and a lot of private benchmarks."* - References external validation selectively: *"a lot of Anthropic blog posts, but also like a lot of other researchers have noted…"* ## Structural Delivery - **Two-target takeaway close**: Splits final advice by audience role — *"if you're a researcher… if you're a builder…"* - **Ends with a personal announcement**: *"This is the first public announcement of it. I'm building a ticket through mechanisms"* — positioning the talk as a launch vehicle. ## Uncertainty Notes - Filler density (*"sort of," "you know," "like"*) is high throughout, suggesting informal, partially improvised delivery rather than scripted. - Several transcript passages are garbled (e.g., *"Opus 4.8 is worse than 4.7"*, *"I'm recording a handshake"*), so some specific claims — model versions, vendor names — cannot be verified as delivered accurately from this transcript alone. # Observation Record — "State of Data" (Sean Cai, Independent) ## Speaker - **Sean Cai**, independent, presenting on data markets; states he speaks on no company's behalf. ## Framing Claims - **Scale of the market**: The annotator-room image of data markets (e.g., Manila labeling rooms, "scale around 2019") is real but "the least interesting part" — roughly **$10–15B per lab per year** by his estimate. - **Thesis**: The scarce, badly priced data is what moves models from **generalist competence to real expertise**; this has been known since ~2024 when Scale was acquired and GPQA datasets were being "spammed." - **Analogy**: Data is to the white-collar revolution what **coal and iron** were to the Victorian age. - **Supply-chain unbundling**: Two years ago, vertically integrated giants (Remarque, Surge, Scale AI) did everything because unit economics demanded it; now specialists outcompete them at sourcing, environments, reward design, and evals. He asserts fragmentation is **permanent, not transitional**. ## Compute/Data/Talent Model - Model improvement is a function of **compute, data, talent** (his "dodge" equation); imbalance produces inefficiency in performance and CapEx. - CapEx is rising exponentially while AI revenues lag; **data is the underfunded leg** — the one converting generalists into experts. ## Type One vs. Type Two Data - **State-based data**: rows in an ERP, final outputs — the 2023 next-token view; mostly personal data under privacy law. - **Process-based data**: trajectories, reasoning traces, sequences of decisions. - **Type one** (non-contrived): pure capture of real workflows (GitHub commits, session replays), minimal reward shaping. - **Type two** (contrived): hired experts manufacturing examples in arbitrary settings. - Claim: Type two was right when models were weak; **type one drives 20→80%** because realism is inherited from real work. A dataset is only available at the frontier — durable supply is a **live business partner**, not dead startups' codebases. - **"Dirty secret"**: "everybody sells type two and bills it as type one." ## Verifiability Framework Cites **Jason Wei's "Verifier's Law"** (ease of training ∝ verifiability). Three axes: 1. **Asymmetry** — decomposability into checkable steps. 2. **Veracity** — consensus on what "correct" means. 3. **Proliferation** — frequency of fresh verified examples. - Coding matured first because GitHub scores high on all three (unit tests, community consensus, commit messages as free reasoning traces). - Predicted maturity ordering: code → search → finance → healthcare/law → cyber, biology, scientific discovery → **taste** (least verifiable). - Biology, security, finance, healthcare, law score low on these axes; examples locked inside enterprise workflows (hence vendors seeking Slack/Jira logs). ## Benchmark Critique - Dominant type-two recipe: experts generate tasks, cherry-pick model divergences, package as a "hard North Star benchmark," then sell data to hill-climb that same benchmark — **"Goodhart's law with a profit motive."** - Market sits in a **"fog of war"**: labs, vendors, enterprises all guessing which data improves models. - Contrived benchmarks test only isolated in-distribution questions; can't test sustained reasoning across long dependent episodes. - Cites Anthropic blog posts and other researchers: **cross-harness/cross-infrastructure differencing** drives benchmark divergence; high false positive/negative rates hidden in headline stats. - **"A single benchmark number is basically one sample from a distribution nobody measured"** — "benchmark psychosis." ## Private Evaluation Example - Claims an internal Val's-AI-style setup with private benchmarks on three finance tasks: **ARR waterfall reconciliation, LBO valuation memo, hedge fund long-short pair trade**. - Findings as stated: **Opus 4.8 worse than 4.7** on several rubrics (over-engineered self-reflection); GPT 5.5 and Opus 4.8 within three points but failing in opposite directions (GPT nails arithmetic, Opus methodology but loses arithmetic). He infers post-training directions from this. - Swiss Army knife analogy: judging a knife broken by cutting cheese with the screwdriver bit. ## Predictive Use of Data Markets - Data-market spending as an **upstream indicator**: Anthropic spent heavily on cybersecurity data in January, biological data in March/April; 2–3 months later came Metis, Cybor, Claude bio/life-sciences offerings. - Checklist: classify tasks on the three verifiability axes; apply long-horizon bar (step length, heterogeneous tools, constraining state transitions, mandatory failure recovery); look for five raw-data signals (sequential decisions per entity, inferable expert actions, independent outcome recording, economically available high-wage work). - **Counterexample — robotics**: modality unsettled (ego vs. teleop vs. UMI); he observes "a huge degree of unsophistication" among robotics data vendors (joke: offers of 100k hours of iPhone video as ego data). - Environment companies, if successful, are **research accelerators** — a boutique industry; venture scalability only if infrastructure serves enterprise app-layer use cases. "Don't die in a modality hill." ## Model Layer / Historical Claims - Claim: **no infrastructure pioneer has held >10% market share long-run** (railroads nationalized; AWS/Google never captured the app layer). Explicit qualification: "I'm not saying that this applies to Anthropic." - OpenAI/Anthropic "fiefdoms" face analogs of railroad-era pressures (anti-distillation, export rules, exclusivity). - **GLM 5.2 surpassing GPT on real-world rubrics** (his claim) as evidence app-layer companies can decouple from the model layer; models aren't fungible like electricity, so neither nationalization nor durable lock-in is the outcome. ## Data Companies → Enterprise - "Successful data companies nowadays are all pivoting to enterprise"; claims an "incredibly large" share of their revenue is now enterprise, "in ways you wouldn't expect." - Missing abstraction layer (five jobs, three enumerated): (1) serve/route small models by latency/performance profiles; (2) manage RL datasets across base-model migration (auto-rerun post-training on base swap); (3) **"Antikythera mechanisms"** — bespoke systems translating messy business context into evals. - Data companies must become **"neo labs"**; durable value accrues to services and app layers. ## Takeaways Stated - **Researchers**: stop outsourcing your definition of realism to the vendors you buy evals/tasks from — "letting the task test writer grade the task." - **Builders**: the moat is not the data but the **pipeline into real-world work** plus infrastructure to keep retraining as models improve. ## Personal Announcement - First public announcement: he is building a new venture ("ticket through mechanisms" per transcript) to help real-world companies monetize data assets and implement **RL-as-a-service** for enterprises. Contact via Twitter/Substack. ## Qualifications & Unverified Elements to Preserve - "$10–15B per lab," "20–30 vendors" mandate figure, and revenue-mix claims are speaker assertions without cited evidence. - Explicit hedge: historical infrastructure analogy "not saying this applies to Anthropic." - Model version names (Opus 4.8, GPT 5.5, GLM 5.2) are as spoken in the transcript; some figures may reflect transcription noise. # Speaker Attribution Record — "State of Data — Sean Cai, Independent / State of Data" ## Video Overview - **Format:** Single-speaker conference talk (AI conference), followed by audience applause. - **Video ID:** `01a15ca8-3984-7e29-85ee-2f7de0e6196c` - **Curated metadata:** None supplied beyond title; title identifies the speaker as **Sean Cai**, affiliation **Independent / State of Data**. ## Speaker Roles | Speaker | Role | Evidence | |---|---|---| | Sean Cai | **Creator / keynote speaker** (independent, per title) | Delivers entire talk; states "I'm here to not deliver an agenda on any company's behalf"; announces "I'm building a ticket through mechanisms… first public announcement of it." | | Audience | **Unidentified attendees** | Non-speaking; laughter and applause noted in transcript markers. | | Narrator / guests | **None present** | No narrator or guest speakers appear in the transcript. | ## Provenance Facts ### Speaker's Self-Description - Claims independence: speaks "not on any company's behalf." - References his own published writing: "a piece called *The TAM is in a vertical*," a chart/equation posted online, and prior January-era pieces "all of which are online." - Final announcement: building a new venture (referred to as "ticket through mechanisms" — transcription of name uncertain) to help companies monetize data assets and implement "RL as a service." ### Referenced Sources & Third-Party Claims (as attributed by speaker) - **Jason Wei** — researcher, cited for "Verifier's Law" via his blog ("whose blog is great. You should read him"). - **Anthropic** — cited for blog posts on cross-harness/cross-infrastructure differencing; claimed data-purchasing pattern (cybersecurity data January, biological data March–April) preceding Claude product launches ("Metis and Cybor" — as transcribed). - **Companies named:** Scale AI (acquisition in 2024 per speaker), Remarque (as transcribed), Surge, Mercari (approaching AI companies for Slack/Jira logs), OpenAI, Google/AWS, OpenAI & Anthropic ("fiefdoms"), GLM 5.2, GPT 5.5, Opus 4.7/4.8, "frontier squeezes / deep squeezes" benchmarks (as transcribed). - **Historical claims:** infrastructure pioneers holding <10% market share long-run; railroad nationalization analogy — asserted by speaker without cited source. ### Unverified / Speaker-Asserted Claims - Private benchmark results (ARR waterfall, LBO valuation memo, long-short pair trade) based on speaker's "internal version of Val's AI and a lot of private benchmarks" — not independently verifiable from the transcript. - Claim that "everybody sells type two and bills it as type one" — speaker characterization. - Vendor revenue claims ("recording a handshake… incredibly large amount of their revenues are enterprise now") — asserted as insider knowledge, no source given. ## Promotional Segments ⚠️ - **Closing segment:** speaker announces his new venture (data monetization / RL-as-a-service) — "This is the first public announcement of it… If you guys want to talk about it afterwards, I'm on Twitter… and Substack." - Recurring self-promotion of his own writing ("I always write a lot on Twitter and Substack"). ## Availability & Limits - **Comments:** No comment availability information supplied. - **Curated metadata:** None provided; all provenance facts above derive from the transcript only. - **Transcription ambiguity:** several proper nouns appear garbled ("Remarque," "Metis and Cybor," "ticket through mechanisms," "Opus 4.8/4.08," "GPT 5.5") — recorded as transcribed, not corrected. Good to give this talk and just want to say right off the bat this is probably going to be a little different from what you've seen so far at AI conference. I'm here to not deliver an agenda on any company's behalf but just to expose a lot of alpha in data markets. But also just tell you what's really going on behind the scenes in a in a very murky landscape where nobody seems to know how Remarque handshake and a lot of these folks actually produce data. So you know, quick reframe before we start when people hear data markets they picture scale around 2019. They picture these rooms of annotators in like Manila labeling images and that's that's real and it's maybe you know, 10 to 15 billion dollars a year per lab but it's kind of the least interesting part. The models work now what scares and badly priced is the sort of data that takes some from generalist competence into real expertise. We knew this since 2024 when Scale got acquired and we were spamming a lot of GPQA data sets. So look, let me start with this framing. Data is to the white collar revolution what coal and iron basically is to the Victorian age and the information age right now. I have this piece called the TAM is in a vertical it's all of labor. And so the supply chain is basically just doing what every sort of industrializing supply chain does it unbundles two years ago when vertically integrated giant like a Remarque or Surge or Scale AI if you if you guys are unfamiliar they're massive data companies. They had to do all of it but that was because that's the only way that unit economics worked in an immature market today. That's increasingly not the case. Specialists outcompete the giants at a lot of steps sourcing the people, building environments, designing rewards, running evals. The fragmentation I would say, is pretty permanent. It's not transitional. And uh quality increasingly does not scale linearly with quantity, which leads to a sort of cottage industry in data land right now, where you have labs literally mandates uh vendor diversification on the scale of like 20 to 30 different vendors because they inherently distrust their ability to scale quality with quantity. So, to orient you, you know, here's here's the sort of argument as it developed through this year. Um January, we saw a lot of industrialization and unbundling. You'll hear me come back to this mechanism I call the Antikythera mechanisms, which is this sort of bespoke systems that translate messy business context into Evals. Uh increasingly important in labs' hungry quest for real-world seed data to set at seed ends. Um I also explored this concept called type one versus type two data, uh contrived versus non-contrived for the more researcher types in the audience, and why the GPQA cell playbook that works in 2024 for data acquisition kind of falls apart on long horizon unverifiable work. Um so, you know, going through all these topics, all of which are online, uh I'll also I'll actually just skip to the more interesting part, but I'm bringing this up just here in case any of the particular topics I talk about are are particularly interesting and you want to dive in more. So, model improvement is a function of three inputs. We all sort of see this uh commonly expressed as dodge, compute, data, and talent. I put together this very uh like rudimentary sort of chart online in one of my pieces and a very rudimentary uh equation just to express the fact that, you know, if there's any sort of imbalance in this compute, data, and talent, then you start seeing uh a sort of inefficiency in producing uh what I call like generalized AI model performance, right? But also, uh we we see a sort of inefficiency in in CapEx spend that it that is a sort of result of this equation right now. Um exponentially increasing CapEx spend, but sort of like um AI revenues are are sort of falling far behind. Uh data is sort of the underfunded leg here. It's the one that sort of turns a generalist model into a real expert. It's the one that's uh actually I believe quite lacking in this equation and thus because of the imbalance penalty parameter concepts that I'm expressing, um presents a whole opportunity. But you know, going back to what I said at the start, what is data actually? So, most people picture state-based data, the rows in an ERP, which is like the final output or saved file. That's kind of the 2023 like next token prediction model um and it's mostly personal data wrapped in privacy law. What is actually available nowadays is process-based data, which is the trajectory, the reasoning trade trace, the sequence of decisions. Um so, it it's kind of what gets a professional to get from a blank page to a finished work output and sort of delineates how the work gets done. Um so, on top of that sits a quality axis. This is the vocabulary. I'll use all talk. Um type type one data is a sort of pure capture of real workflows like GitHub commits or session replays with minimal reward shaping by non-experts and type two is contrived data, where you sort of hire experts, you sit them in an arbitrary setting, you have them manufacture examples. Type two, right place to start models when they were reading at a first grade level, I think anybody in the world could teach them as a third grade teacher. But type one is what gets you from 20 to 80% so to say cuz the real the realism is inherited from the work itself and the structural reason why it matters is the data is kind of the most appreciable asset there is. Um a data set is only available in so far as frontier data markets as the frontier moves. So, the only durable supply of it technically is a live business you partner with, not a dead startup's code bases like so many data companies out there are buying today. The dirty secret of the industry though is that everybody sells um type two and bills it as type one. So, now let me talk about the central axis of how you can think about the sort of verification and deciding which application layer domains are are like maturing first and why. Uh you know, there's a reason why cloud science came out so far um you know, now in the future after models got good before a lot of other application layer advances. So, Jason Wei is a researcher online whose blog is great. You should read him. He has this law. It's called Verifier's Law. The ease of training a model to do a task is proportional to how verifiable the task is. So, I break verifiability to three axes here. And once you have them, a huge amount of this market stops being mysterious. First, asymmetry of verification. How hard is it to decompose a task into checkable steps? Veracity of verification. How much consensus is there about what correct even means? And then thirdly, proliferation of verification. How often does the real world sort of hand you fresh examples of verified work? If you think about like coding as the first mature AI app layer market, that's really no accident because we were blessed to have something called GitHub from web 2.0 which solved all three of these at once. Unit tests, they sort of give you objective decomposable correctness. The community agrees on what working code means generally. And there are effectively infinite public examples with these commit messages as basically free reasoning traces. So, they score high, high, and high on these three axes. Now, look where the money is trying to go now. Biology, security, taste, finance, health care, law. These sit pretty low on veracity, pretty low on verification, and the verification examples are sort of like locked all in these enterprise workflows. No web 2.0 system ever captured really or very sparingly few. Uh that's why, you know, probably maybe you've been approached by Mercari to buy your Slack logs or your Jira logs as of late if you work at an AI company. And that's the whole game, right? It's it's predictive. It's not just descriptive. Classify any professions like tasks on these three axes, and you can sort of tell which markets mature most. So, it's no surprise that after code, we went to search, and after search, we went to finance, and after finance, we went to healthcare and law. And after healthcare and a law, well, I'd say cyber, biological, uh and scientific discovery. And maybe even taste, which is probably the most unverifiable out of all these. So, a verification is a sort of bottleneck. Um let me talk about why the industry sells a lot of snake oil here, and why most benchmarks you see are are sort of quietly fake. Um the dominant type two recipe is like let's hire domain experts. Um let's have them chat use chat to generate plausible tasks. Let's have them solve those tasks, and let's cherry-pick the ones where the model diverged, and let's package them as a hard North Star benchmark. And then, this is the perverse part, let's sell the data to hill climb that same benchmark. It's basically um and maybe some of you guys here in SF have heard this a little too much. It's Goodhart's law with a profit motive. Basically, the moment your measure like becomes a target, and then the target is set by people who aren't true domain experts, it stops sort of measuring anything real. And so, the whole market, uh I like to call it sits in a sort of fog of war. Labs, vendors, and enterprises, they're all kind of guessing which of the data actually improves a model. Nobody can see this clearly. There's a structural tell. Contrived benchmarks, they only ever test a single isolated in-distribution question. They can't test whether a model sort of sustains like correct reasoning across a long dependent episode because these tests were all sort of meant to be solved in isolation. Uh and may I spent a lot of time in this. Um increasingly, uh in a lot of Anthropic blog posts, but also like a lot of other researchers have noted that cross-harness differencing and cross-infrastructure differencing is the primary cause for a lot of benchmark divergences in performance. A lot of you are probably very familiar with like the frontier squeezes of the world and the deep squeezes of the world benchmarks. Um there are issues that all of these benchmarks have uh related to that specifically. In particular, just high false positive and false negative rates that are not immediately obvious when you look at the above head stats on the benchmark. So, a single benchmark number under a single scaffold is like basically one sample from a distribution who's basically with nobody measured. And that's why there's so much what I call benchmark psychosis today. It's like a it's a pretty noisy sample. So, um you know, really briefly on an example I took I I always have like I basically have an internal version of Val's AI and a lot of private benchmarks. Um so I I took three finance tasks from three real-world vendors. Um like an ARR waterfall reconciliation and LBO valuation memo and a sort of like long short pair trade for hedge fund uh trading. So, it's a long horizon non-verifiable finance tasks, relatively robust like deterministic verifiers paired with LLM as a judge uh applied correctly. So, um when you actually run these, I think you'll you'll notice very clearly Opus 4.8 is worse than 4.7 on a lot of these rubrics. Uh you'll you'll start noticing things if you actually do rubric analysis that the 4.7 to 4.8 sub over-engineered self-reflection. Um you'll you'll notice that uh GPT 5.5 and Opus 4.08 score within three points on the same task. They fell in like exactly opposite directions, whereas GPT nails the arithmetic um but Opus nails the the methodology but loses arithmetic, which is to say uh it's it's clear if you actually do very good agnostic benchmarking, um you you can tell where the post-training directions for a lot of these teams went. And subsequently, the it informs, I think, like a lot of the data that that comes for this. Um so, look, you know, going back to the start, a single benchmark number is a sample from a distribution nobody measured, and it's basically like taking a Swiss Army knife and using the screwdriver bit to cut cheese and like concluding the knife is broken. Um so um you know, the the receipts from the last slide, this this goes into an RL environment report that is sent to labs. Um but no single leaderboard number ever shows you this part of um you know, data, which some of the most sophisticated RL environment companies, some of which are actually here at this conference, um would would show you this. So uh how do we read sort of which domain is next instead of chasing hype, and you can actually use this as a proxy data markets as an upstream indicator of what next application layer products that labs will come out with. In in January, uh Anthropic was spending a lot on cybersecurity data from new vendors. And in March and April, they were spending a lot on biological data um from from associated vendors, and what do you know happened like 2 to 3 months after? Well, Metis and Cybor and Claude bio / life sciences today. Uh So if you want to use this checklist I use, classify the professional's tasks into three axes, apply to long horizon bar, the the ones real labs use, um enforced step length heterogeneous tool calls that aren't interchangeable, state transitions that genuinely constrain future actions, mandatory failure recovery. Uh you'll notice a pretty mixed bag of of how long horizon is even defined from the vendor perspective in terms of specs. Um and then three, you want to look at the raw data for five signals. So, sequential decisions against like one entity, an inferable action expert action per step, outcomes recorded by independent parties. Um but moreover, just economically available high-wage work. So, uh so subsequently, one one counterexample to keep you honest, like robotics, the modality is not settled. Um ego versus teleop versus UMI, but moreover, I I find just generally a huge degree of unsophistication within robotics data vendors today. A huge degree of unsophistication. Yeah. Uh I don't know. Some of you guys are probably robotics researchers in a crowd. I don't know how many times like people have come up to you and they're like, "Oh, here's like 100,000 hours of like iPhone video from my friends in India. Do you want to buy this for ego data?" \>\> \[laughter\] \>\> Um so uh in a in in that sort of domain your vendors uh choices entangled with an unsolved research question. At the end of the day, you have to realize like all environment companies if they actually succeed, they're more so research accelerators. It's a boutique industry. If it's venture scalable, it's because the infrastructure they're building agnostically in house helps an enterprise application layer use case rather than assuming that data markets today will stay as they will forever. So, don't die in a modality hill, basically. So, so one last exhibit just for my work. This is the map under everything I just described. Look, this share of the white-collar work is sort of on the vertical axis. This task horizon is on the horizontal. Short horizon is a is very addressable right now, but you think about this long tail on the right side, this deep dependent long horizon work. That's where the real economic value and data build out sort of both lives. And this threshold line that kind of kind of moves rightward every time somebody builds a real-world data pipeline. Um now I want to talk a bit more about the model layer. Um it changes who needs to build what. So, historical fact, no pioneer of an infrastructure technology has actually held more than 10% of the market in the long run. Um I'm not saying that this applies to Anthropic, but you know, I'm just those who forget history are condemned to repeat it. Railroads built company towns. They charged tyrannical rents. They got nationalized as soon as the automobile moved in. AWS and Google that consolidated the infrastructure layer and still never captured the application layer. And right now, OpenAI and Anthropic are carving out these fiefdoms, but like all these pressures from anti-distillation, export appeals, enterprise exclusivity. They're basically the equivalent of real-world brands in their road. Right? Like the automobile in some cases has already arrived like GLM 5.2 surpassing GPT on a lot of real-world rubrics. Um is is is pretty hard proof that a lot of app layer companies can decouple themselves from the model layer. And because models differ on efficiency and modality, they're not fungible like electricity. So, it's not it's not exactly leading to nationalization, but it's definitely not heading to durable lock-in either. And the whole question on what to build next hinges on whether a general enterprise can decouple models from foundation model labs. Um But luckily, you know, we for a lot of data companies, this is actually where they're headed. I came here to give a talk on data markets. I'm here to tell you that that successful data companies nowadays are all pivoting to enterprise. I maybe shouldn't say this in a public setting, but I'm recording a handshake. People don't notice an incredibly large amount of their revenues are enterprise now. Enterprise in ways you wouldn't expect a data business to do. Once enterprises stop renting out labs intelligence and starts owning their own, you need an entire abstraction layer that doesn't exist yet. There's like five jobs serve and route small models targeted by costly latency and performance profiles. Um And uh I to manage your RL data sets instead of like um across base model migration. So, when you swap to a new open source base, you rerun post training automatically instead of starting over. Three, the Antikythera mechanisms that I talked about before. Um and in in the interest of time, happy to talk more about like emerging infrastructure needs afterwards if you want to. So, let me bring it all together. I think data companies all realize that they have to be neo labs. Data businesses do not stay data businesses because the durable value sort of accrues to the services and app layer of actual work. So, two takeaways, you know, if you're a researcher, stop outsourcing your definition of realism to the same vendors you buy your eval's and tasks from. Um, that's kind of just letting the task test writer grade the task. And if you're a builder, your moat's not the data, it's the sort of pipeline into real-world work. Plus the infra to keep retraining on it as the models improve underneath you. I'll close on this. Uh, I'm building um, I'm working on something new. This is the first public announcement of it. I'm building a ticket through mechanisms. I'm working with a lot of real-world companies to monetize their data assets, but moreover, help implement RL as a service with a lot of the enterprises in the world, mitigating a lot of the pitfalls of a lot of the companies I mentioned. If you guys want to talk about it afterwards, I'm on Twitter. I always write a lot on Twitter and Substack and um, I'm around afterwards, too. Thank you. \>\> \[applause\]