Hrishi Olickel (Southbridge AI) explains why data transformation requires more than a universal schema or a confident LLM agent, using failures like a pipeline that dropped minus signs from loan amounts — turning a net $784,000 into $5.3 million of debt — with multiple models agreeing on the wrong result. He proposes a just-in-time ontology: build the transformation when the question is known, preserving context and evidence for reuse. An Ohio campaign finance entity resolution project illustrates separating models that design pipelines, models that execute individual jobs, and models that optimize the system, with emphasis on definitions, held-out cases, disagreement checks, bounded budgets, and reusable, independently checked work segments that let the pipeline trade among cost, speed, and accuracy without losing evidence. Timestamps from 00:00 to 18:14 are available in the original record. # Video Identity Observations ## Source Overview - **Video title:** *Just-in-Time Ontologies: What Cheap Intelligence Means for Data — Hrishi Olickel, Southbridge AI* - **Format:** Conference-style technical talk by a named founder, presenting company work alongside conceptual arguments. ## Self-Description Evidence (Creator Claims) - The speaker explicitly describes his company's identity and ethos: > "We are best described as obsessives. Obsessives who embed intelligence into pipelines." - Company positioning claims include: - Building **repeatable, self-optimizing data pipelines**: "Pipelines that can scale... that can mature in cost, performance, and speed all on their own." - Track record claim: "In just 2 years, we've gone from figuring out how to make early 3.5 turbo level models reliable on things like health care, aging, mining research, to building Hank Wave, our open-source harness for data work." - An admission of a revised prior belief: "I talked about a single universal schema when I founded Southbridge 2 years ago... This is why I was wrong. There is no universal schema." - Closing affiliation statement: "And at Outpace, we're working as hard as we can to make that future a reality" — suggesting a company/brand name (note: transcript also says "Southridge" at one point, an apparent variance from "Southbridge" in the title). - Open-source posture: promotes **Hank Weave** ("our open-source harness for data work") and invites audience use ("I'll leave the skill up if you'd like to try it"). ## Claimed Scope - The talk claims to cover a defined technical agenda: "why single ontologies always fail, the three roles models play in pipelines, intersecting model priors for better results, and Pareto serving." - Broader thesis claimed: cheap intelligence triggers a "paradigm shift" in data transformation, because "all of our data infrastructure today is built on the idea that transformation is expensive." ## Intended Audience (Observable Presentation) - **Data/ML engineers and technical practitioners**: assumes familiarity with entity resolution, AB testing statistics, schemas, reasoning models, and token costs. - **Enterprise data decision-makers**: repeatedly addresses enterprise pain ("In almost every enterprise... most of the data being recorded has little idea of what it's going to be used for"). - **AI-builder community**: references agent runtimes ("cloud code and pie"), prompting, agentic patterns, and offers screenshot-ready advice ("I'll leave this up so you can screenshot it"). ## Demonstrated Presentation Patterns - **Case-study-driven credibility**: detailed Ohio campaign finance entity resolution results (203,317 cases, $795 actual vs. $10,000 projected cost) with downloadable artifacts: "you can download it for yourself, prompts and everything." - **Error-transparency as a motif**: catalogs model failures (in-kind parsing, date formats, sign conventions, "$784,000 became 5.3 million dollars of debt"). - **Conceptual vocabulary building**: coins terms like "just-in-time ontology," "Pareto serving," "burrito surfing," "codons." ## Boundaries - This is a single conference talk; channel-level conclusions (publishing cadence, recurring formats) cannot be inferred from it alone. Company naming shows minor inconsistency in the transcript ("Southbridge" / "Southridge" / "Outpace") and should not be resolved by inference. # Delivery Observation — Hrishi Olickel, "Just-in-Time Ontologies" ## Overall Register The delivery is **conversational-technical**: an insider explaining a working system rather than a formal academic talk. Sentences are short and declarative, and the speaker frequently addresses the audience directly as a collaborator ("Say, I ask you to write down the temperature. Say, I ask you to write down the temperature" → "The source schema is whatever you have access to"). ## Tone - **Opinionated and self-correcting.** The speaker openly retracts a prior position: > "This is why I was wrong. There is no universal schema." - **Insider-confident, but hedged at edges.** Strong claims are common ("I believe we've solved the final bottleneck in transformation"), punctuated by occasional hedges ("Not quite," "To be fair," "I'm not going to say who"). - **Dry humor, understated.** > "Not because you're not going to get a result, but because you will." And: > "Sometimes you can solve this just by not vibing, by checking results and applying human intuition." ## Vocabulary and Signature Terms - **Coined or branded phrases recur throughout:** "just-in-time ontology," "Pareto serving," "failure surface," "burrito surfing," "Hank Weave," "write once, run once / write many, run once / write many, run many," "decision models," "cross-model prior intersection." - **Concrete numeric evidence is woven into rhetoric:** "$10,000" vs. "under $800," "$795 on an $800 budget," "203,317 cases," "over a dozen architectures," "6 billion tokens," "99.4% of transaction dates." - **Colloquialisms mixed with technical jargon:** "noble defects," "thicker tokens," "yes, sir," "monument scale immediately invoked functions" (transcript garbled; likely "monster-scale"). ## Teaching Habits - **Concrete everyday analogy before abstraction.** The temperature example carries the entire four-schema argument; the leafcutter ant colony illustrates specialized workers: > "A leafcutter ant colony, for example, runs on four different sizes of worker." - **Structured enumeration as a spine.** Nearly every section is numbered: three roles models play, four schemas, four things models are missing, six stages. The speaker announces the roadmap up front ("we're going to cover why single ontologies always fail, the three roles models play..."). - **"Screenshot-able" moments, verbally flagged:** > "I'll leave this up so you can screenshot it." - **Case study used twice deliberately.** The Ohio/Jeff entity-resolution story appears early as a hook and returns in depth later — a clear intentional repetition. - **Rhetorical questions as transitions:** "What if that wasn't true anymore?" / "So, can models do this natively now? Not quite." - **Callback structure.** Promised threads are paid off by name: the fourth schema ("which we'll get to when we talk about priors"), "burrito surfing and we'll cover it at the end." - **Authority citation used sparingly but pointedly:** Archimedes ("Give me a place to stand, and I will move the Earth") and Adam Smith ("as Adam Smith would say"). ## Recurring Sentence Patterns - **"This is [the/what]..." definitional pivots:** "This is the schema imposed on you by storage," "That's the other half of what I mean by just-in-time ontologies." - **Anti-climax corrections:** "But buying isn't the primary reason," "But here's where the money went." - **Contrast pairs as the dominant argument form:** source vs. decision, doing vs. checking ($128 vs. $545), expensive slow humans vs. cheap fast models, big models vs. pipeline workers. ## Stable Patterns vs. Isolated Moments - **Stable-looking (repeated across the talk):** enumeration discipline, the same case study anchoring the argument, coined terminology reused consistently, numeric receipts as persuasion. - **Possibly one-off:** the personal anecdote about a previous job ("my job was to make sure that certain numbers just did not exist ever") and the closing company plug ("at Outpace, we're working as hard as we can"). One mention each; **uncertain** whether these are habitual traits of the speaker's style. - **Caveat:** transcript contains apparent artifacts (e.g., "Hank Wave" vs. "Hank Weave," "Jev" vs. "Jeff," "Outpace" vs. earlier "Southbridge"); naming consistency could not be fully verified from the transcript alone. # Observation Record: "Just-in-Time Ontologies: What Cheap Intelligence Means for Data" — Hrishi Olickel, Southbridge AI ## Core Thesis **Speaker:** Hrishi Olickel (Southbridge AI) **Central claim:** All modern data infrastructure assumes transformation is expensive; the final bottleneck in transformation — judgment — has been solved by accessible general intelligence, enabling a paradigm shift he calls **just-in-time ontologies** — building/merging schemas at decision time rather than ahead of time. - Opens by citing an article from "Hannes at Duct Tape B" asking whether a decade was lost chasing distributed architectures premised on slow, expensive compute/storage. - Frames his position: *"Capability in every era defines architecture and tooling, and the tooling lags behind consistently."* - Anchor example: Ohio campaign finance entity resolution — a pipeline projected at ~$10,000 was completed statewide for **$795** within 10 days of a new model (referred to as "Jev"/"Jeff") launching, after his system tested over a dozen approaches. ## Why Single Ontologies Fail: The Four Schemas **Claim:** The popular explanation (lack of buy-in) for failed migrations is "100% true" but **not the primary reason**. Even with instant migration, "there is no super schema" — there are four schemas: 1. **Source schema** — whatever the information source has access to (illustrated via temperature: phone vs. desk sensor vs. watch; even the recorder's patience is part of it). 2. **Storage schema** — limits of what can be recorded (you can't record device firmware or time to the second). 3. **Decision schema** — the context of the decision the data serves (comfort monitoring vs. weather mapping vs. equipment siting each require different temperature data). 4. **Model prior schema** — described as "a recent addition," the beliefs the model brings to the task. **Key qualification he makes about himself:** He admits his own past claim was wrong — he advocated a "single universal schema" when founding Southbridge two years ago, and now states, *"There is no concept of a customer that's divorced from the question that you want to answer."* - Enterprise data is recorded without knowing its future use; the four schemas converge "only at the moment that the decision is being made." - Transformation = merging the recording context with the decision context; doing this at decision time = **just-in-time ontology**. ## Three Regimes of Data Work | Regime | Description | |---|---| | **Write once, run once** | Single-use jobs (joins, mining one answer), torn down after | | **Write many, run once** | Many workloads for one goal, e.g., large migrations; each verified at source | | **Write many, run many** | Repeatable pipelines built/maintained over time — his "bread and butter" | ## Three Roles of Models in Pipelines **Claim:** Coding agents conflate three distinct roles in one context window; they should be separated: 1. **Pipeline builders/designers** — want slower models, "heavier, thicker tokens," reasoning-heavy (he names "Opus or up"). 2. **Pipeline workers** — want cheap, fast, specialized, repeatable models with "noble defects." Analogy: leafcutter ant colonies run on four worker sizes. 3. **"Hill climbers" / servers** — models that self-optimize: try new things, vary budgets, debug, take notes. He calls the process **"burrito surfing."** ## Anecdote: "Can We Make This Decision Data-Led?" - At his last job, the question was appropriate at thousands of vessels but misapplied at 50 users: *"you will get a result"* from A/B testing 50 users, which is statistically misleading. - His stance: *"the only real option was to not produce a number at all, because the moment a number exists, it argues for a result."* - **Qualification on today's tools:** interactive agents "will always give you an answer," so mistakes compound — "a moment in chat can be a broad nightmare forever just waiting to be discovered." ## Jev Entity-Resolution Results (Ohio Campaign Finance) - Three initial designs on ~200 held-out families: cheap reasoning model alone matched "Fable" at ~2% of cost; Jev-first answered correctly but confidently rejected all 9 household links; rewording questions to specify "what counts as enough evidence" flipped Jev from no to yes **without changing models or thresholds**. - Over a dozen qualification rounds statewide, fixing definitions (committee, filer, cohabitation). - All **203,317 cases** labeled with what the answer rests on and how far it was checked. - **Cost breakdown:** $795 total vs. $800 pre-set budget; checking cost ~4× doing ($545 second opinions/audits vs. $128 first answers). Next state's audit budget expected "close to zero." - Agent ("4A" on Astra, later Opus) found most approaches incrementally: 10 days, 12+ architectures, 4,000+ runs, ~90,000 Jev calls, ~6 billion tokens. - Learnings written back into a version-controlled library of "agentic parts" (Hank/Hank Weave). ## Model Priors as the Fourth Schema **Claim:** Bigger models can be *worse* — they hold priors and argue for them more vehemently. Examples of prior-induced errors: - "In-kind" (capital I, hyphen) vs. expected "in_kind" — 15,658 contributions counted as ordinary money; loan repayments vanished similarly. - Four date formats tried vs. the file's fifth — 99.4% of dates empty, but row/dollar totals matched so the agent moved on. - Negative loan amounts "corrected" to absolute values — $784K net became $5.3M of debt, "validated by multiple models." He says this is becoming **the largest class of error** he sees across companies doing data work. **Mitigations (with stated limits):** - **Human checking** = intersecting human priors with model priors; effective but humans are "expensive, slow, non-deterministic," and "hate interruptions." - **Cross-model prior intersection:** Jev and Luna agreed on 194/200 families; **none of the shared answers was wrong** — all errors sat in the 6 disagreements. Models from different pedigrees disagree more. - **Qualification:** distilled models (he declines to name the company) are an exception — they share priors. - **Residual limit:** even cross-family models share species-level priors (four families produced identical wrong totals due to a shared sign convention), leaving a persistent error class only addressable by listing and specific solutions. ## Four Prerequisites for Self-Optimizing Systems 1. **Stable deployment surface** — Hank Weave harness (claims it's the most mature/open for headless data work, "to be fair" acknowledging alternatives exist). 2. **Bounded experiments** — cost, time, access controllable as resources. 3. **Know-how and methodology** — new models can apply two years of agentic patterns token-efficiently. 4. **Failure surface** — "the hardest to put into words"; quotes Archimedes: *"Give me a place to stand, and I will move the Earth."* He claims: "we haven't met a problem or a constraint that we couldn't iterate ourselves to solving" given a stable stack and failure surface. ## Six-Stage Process Inspect/split the job → define failure surface (manually checked baselines, schema checks) → minimal bounded pilots → move into "codons" (reusable bounded work segments) → deploy to production → **surf the Pareto frontier** (swap models for cheaper/faster, tighten success requirements, compare against known intermediates). ## Closing Position - **Apology/refutation:** *"The answer is there is no ontology. Not a universal one, at least. There never was one."* - **Reframe:** this makes every ontology/schema valid "as long as you can label and preserve the context around it." - Predicts a future where cheap intelligence makes "any information, in any format" maximally useful; notes work continuing at "Outpace" (name as stated in transcript). ## Tensions Preserved - He founded Southbridge on a universal-schema thesis he now explicitly disavows. - He praises self-optimizing agents while warning interactive agents always produce answers that "argue for a result." - He notes bigger models are sometimes worse at priors despite being more capable generally. # Video Attribution & Provenance Record **Video ID:** `01a15ca6-73a0-7abe-bd54-192f159fe30e` **Title:** *Just-in-Time Ontologies: What Cheap Intelligence Means for Data — Hrishi Olickel, Southbridge AI* --- ## Speaker Identification ### Primary Speaker - **Hrishi Olickel** — identified via video title metadata as affiliated with **Southbridge AI**. - Role: **Creator/presenter** of the talk (conference-style technical presentation). - No curator metadata is supplied beyond the title; role attribution beyond "presenter associated with Southbridge AI" is **uncertain**. ### Other Voices - None identified. The transcript is a single continuous monologue (marked by ">>" speaker-change glyph at the start only). - No guests, narrators, or audience interaction are detectable from the transcript. --- ## Self-Identified Roles & Context (per transcript) - Speaker states he **founded Southbridge** ~2 years prior: *"I talked about a single universal schema when I founded Southbridge 2 years ago."* - Speaker references a **"last job"** (prior employment, unnamed): *"a story from my last job"* — platform with "thousands of vessels" and ~50 users. - Closing line references **"Outpace"**: *"at Outpace, we're working as hard as we can…"* — **unresolved inconsistency** with "Southbridge AI" branding in the title; both entity affiliations are recorded as stated, relationship unclear. --- ## Referenced Sources & Third-Party Claims ### Cited External Parties - **Hannes at "Duct Tape B"** — opening reference to an article questioning distributed architecture assumptions. (Name/company likely garbled in transcript; recorded as-is.) - **Models referenced by name:** Jeff/Jev, Luna, Fable, Opus, Claude Code ("cloud code"), "Astra" — entity resolution workloads reportedly run "90,000 calls to Jeff" and "roughly 6 billion tokens." - **Archimedes** — quoted: *"Give me a place to stand, and I will move the Earth."* - **Adam Smith** — invoked on specialization; **no direct quote**. - **Jev** — described as launching "on a Tuesday" during the entity resolution work. ### Internal / Promotional Claims ⚠️ The following are **self-promotional segment** content and should be flagged as vendor claims, not independent sources: - **Hank Wave / Hank Weave** — described as their *"open-source harness for data work"*, claimed to be *"the oldest, most stable, and the most open"* for headless data work. **Promotional.** - **Southbridge / Outpace services** — self-optimizing pipeline systems, "burrito surfing" methodology, agent "4A" runs. **Promotional.** - **Downloadable materials:** speaker states results *"can [be] download[ed] for yourself, prompts and everything"* and that a "skill" for Hank Weave is left up on screen — availability of links is asserted but **not verified** by this record. ### Key Quantitative Claims (speaker-reported, unverified) - Ohio campaign finance entity resolution: **~800,000 rows / 203,317 cases**, statewide cost **$795** vs. prior **$10,000** projection; checking cost $545 vs. first answers $128. - 10-day agent run, "over a dozen architectures," 4,000+ runs, 90,000 Jev calls, ~6B tokens. - Error anecdotes: 15,658 misclassified in-kind contributions; 99.4% empty dates; $784k vs. $5.3M sign-convention error. - 200 held-out families: Jev and Luna agreed on 194; all errors in the 6 disagreements. All figures are **attributable solely to the speaker**. --- ## Availability & Provenance Limits - **No curated metadata supplied** beyond title — no description, links, or speaker bio to confirm identity or affiliations. - **No comment section data** provided. - **Chart/data downloads and skill links** are referenced visually/verbally but no URLs are present in the transcript. - Transcript contains **transcription artifacts** ("Hank Wave"/"Hank Weave," "Jeff"/"Jev," "Outpace" vs. Southbridge) — names recorded with variants, not resolved. - No external verification of any cited source or benchmark claim is available within the source material. --- ## Provenance Summary | Element | Status | |---|---| | Speaker | Hrishi Olickel (Southbridge AI) — via title only | | Format | Solo conference talk / monologue | | Promotion flagged | Hank Weave, Southbridge/Outpace services | | External sources | Hannes article (Duct Tape B), Archimedes quote | | Third-party claims | Model benchmarks, cost figures — speaker-reported only | | Comments/links | Not supplied | | Unresolved | "Outpace" vs. "Southbridge" affiliation; name spelling variants | \>\> So, I really love this article from Hannes at Duct Tape B, where he asks if we lost a decade chasing distributed architectures that presumed compute and storage were slow and expensive. Capability in every era though defines architecture and tooling, and the tooling likes behind consistently. We first made compute cheaper than storage, and then we gave it to everybody, and then we made it faster. And every single time architectures undergo this massive change. All of our data infrastructure today is built on the idea that transformation is expensive. What if that wasn't true anymore? I believe we've solved the final bottleneck in transformation, general intelligence or judgment, and we've made it accessible on tap all along the Pareto frontier. Which puts us right on the verge of a paradigm shift. So, in this talk, we're going to cover why single ontologies always fail, the three roles models play in pipelines, intersecting model priors for better results, and Pareto serving. How today's models can tune a workload for cost, speed, and accuracy all at the same time. Here's a quick example. When Jeff launched, we already had an entity resolution pipeline running, working out Ohio's campaign finance records and whether they belong to the same person, company, or committee. It was projected to cost about $10,000. Our systems took that new model, worked out where it fit, tested over a dozen approaches, and integrated it. 10 days after that first call, we had final ground truth for every donor case in Ohio under $800. That is the power of Pareto serving and just-in-time transformations. So, let's get into it. Ontologies are the enterprise version of mind maps. It's this carpet that sucks us all in, including me. I talked about a single universal schema when I founded Southbridge 2 years ago. The popular conception, and it's not wrong, it's just small part of the picture, is that migrations and ontologies fail because of a lack of buying. And they're abandoned halfway through, that organizations and people never switch over. All of that is true. 100%. But buying isn't the primary reason. Say, if you could integrate and migrate all of your information into a single super schema in the blink of an eye, you still have to contend with the truth that there is no super schema. There actually are four schemas. The first one belongs to the source, wherever the information comes from. I take temperature as an example. Say, I ask you to write down the temperature. The source schema is whatever you have access to. Your phone gives you your region's temperature, sensor on your desk, which is less accurate, but it's more local, or your watch, say. But even your patience in following through on my request is part of that source schema. The second one is the storage schema. There's only so much you can write down. You can't write down the firmware of the device you're using. You can't even write the time down to the second because you couldn't finish writing all of that inside of a second. This is the schema imposed on you by storage. The third one belongs to the decision. Why do I need you to write the temperature down? Maybe you're figuring out what makes you comfortable, in which case I need temperatures from you spread throughout the day, and I need to make sure that it's you that's doing the measuring. Maybe I'm building weather migration maps, in which case I don't care who does the measuring, but I care a lot about your exact location, the time, the device you used, and whether that's changed. Or maybe I'm deciding whether to install sensitive equipment and where, and you yourself become this variable that I need to cancel out. There's a fourth one though, which we'll get to when we talk about priors. It's this recent addition. The true problem is that you will never know all four schemas at the same time except at the moment that the decision is being made. In almost every enterprise and every home, most of the data being recorded has little idea of what it's going to be used for. Take a customer, the first box in every ontology. To building, a customer is whoever pays. To supported, whoever calls. To compliance, it's whoever we're on the hook for. And the only time you know is when the decision is known. Should we refund this order? Can we extend this account more credit? Suddenly, what you need to know, how and to what degree, becomes extremely clear. This is why I was wrong. There is no universal schema. There is no concept of a customer that's divorced from the question that you want to answer. And pulling these schemas together is the process of transformation, of making information actually useful. Taking the context that it was recorded in and merging it with the context of the decision. And that takes discernment. Doing that merge when the decision space is known instead of ahead of time is what I mean by a just-in-time ontology. So, we'll come back to discernment with a story from my last job where like most of my job was to make sure that certain numbers just did not exist ever. Before that, a little bit about me and us. We are best described as obsessives. Obsessives who embed intelligence into pipelines. Pipelines that can scale, that can run over and over again to take work truly off our hands, and pipelines that can mature in cost, performance, and speed all on their own. And most of the work we do in data breaks into three regimes. The first is write once, run once. If you do any kind of development, think of these as monster scale immediately invoked functions. They're built for something single use, like joining a data set, mining information, or answering one specific question, and then they get torn down. The second is write many, run once, where lots of smaller and larger workloads are built for one large common goal. The closest thing you might know is like large migrations. Years of records moved out of systems nobody fully remembers. Lots of workloads built for one rule and verified individually at source, checked against the others because all of it only happens once. The third, though, and this is bread and butter, is the most common, is write many, run many. These are repeatable pipelines that are built and maintained over time as we add more information in. And at Southridge, we've moved so much work off the main thread and into these repeatable systems, from benchmark design, tracking, coding, to making subtitles for movies that we want to watch. Almost everything we do in the modern world is a transformation of data, or it's bottlenecked by how hard it is to transform data. And in all of these cases, our job is simple. Make sure people can rely on the results. In just 2 years, we've gone from figuring out how to make early 3.5 turbo level models reliable on things like health care, aging, mining research, to building Hank Wave, our open-source harness for data work, and then using that to get models to work together at terabyte scale. Today, we have self-optimizing systems. You give them the problem, a way to fail safely, a place to work, and they find the shape of the solution. They pick the right models, they run the pilots, they scale up, and they Pareto serve their way to the best option they can find. So, story at my last job, the question we heard most often was, "Can we make this decision data led?" And sometimes that was the right question when we had thousands of vessels on the platform. The problem is that we also asked it when we only had 50 users. And if you know statistics, you can explain really well why AB testing 50 users is a bad idea. Not because you're not going to get a result, but because you will. So the only real option was to do not produce a number at all because the moment a number exists, it argues for a result. Today though, it is worse. Interactive agents will always give you an answer. They'll give it to you quickly. You ask for a transformation, whatever, and you get a contra and yes, sir. What this means is mistakes add up quickly. Moment in chat can be a broad nightmare forever just waiting to be discovered. The primary reason that this happens is that intelligence, as it relates to data work today, occupies three separate roles that coding agents kind of mash into one context window. Let's pull those apart because it'll help a lot when we get to decision models because they fit into exactly one of these three. The first one is building and designing the pipeline, which is breaking down the problem, writing the early specs, authorizing spend. The second one, and this is usually confused with the first and given to the same models, is working inside the pipeline, actually doing the work, reading the PDF, writing to databases, checking results. Before we get to the third one, I want to make it clear that these two roles pull in very different directions. The first one wants slower models with heavier, thicker tokens, models that can think more, walk around a problem and see the angles, digest documentation, understand how these pipelines get built, and predict how something will do before it even starts, right? Opus or up. The models you want inside pipelines though are the opposite. You want them cheaper, faster, specialized, good at following instructions, repeatable workers with noble defects. You see the same thing in nature. A leafcutter ant colony, for example, runs on four different sizes of worker, right? Small, medium, high, extra high. As you watch any system that you design, review, or hear about, listen for what slots in where in this layer cake. Makes a big difference. Because we had this distinction, bringing in Jev in this newly exploding class of decision models was very easy cuz they're workers. They're built for quick contact with information and they're deployed at the bottom end of large workloads. So, third category, though. This is something we call the hill climbers or servers. Models that can take a first one and self-optimize. Given the right knowledge, instructions, and an incredibly stable surface to stand on, they can try new things, vary budgets to limit cost exposure, debug quickly, take notes, surf their way not just to the right solution, but the optimal one. We call that burrito surfing and we'll cover it at the end. But, first, let's talk about our experience with Jev. Jev launched on a Tuesday when we were deep into an entity resolution pipeline. You were working on donors in Ohio's campaign finance. And at this point, a lot of people have seen this chart. We talked about the results before and adapted some of the ongoing work to the data set. So, you can download it for yourself, prompts and everything. What we didn't talk about is everything behind that one arrow and what happened when you ran the same idea on the whole state. Here all of the architectures we tried. We started with three designs on a small development set. A cheap reasoning model on its own, Jev with a reasoning model for anything it wasn't sure about, and a reasoning model proposing answers with Jev checking them. On about 200 health out families, the cheap reasoning model on its own did as well as Fable for about 2% of the cost. Jev first got the core answers right, but it missed nine household links because it had confidently said no to every one of them. So, we rewrote the questions. Without touching the models of the thresholds, the household question now said what counts as enough evidence, and Jeff went from no on all 13 development questions to yes. And as we scaled it statewide, we ran over a dozen rounds of such qualification. Each round fixed a definition, what a committee is, what a filer means, when two people live at one address, what does that mean? Every one of Ohio's 203,317 cases now has a label that says what the answer rests on and how far it was checked. This is the hierarchy we discovered for this specific problem. And here's everything we learned about putting decision models into pipelines. I'll leave this up so you can screenshot it. But here's where the money went. The whole statewide job came to 795 on an $800 budget that was set ahead of time. Checking cost about four times as much as doing, 545 on second opinions, settling disagreements, and the audit against 128 for first answers. Before Jeff, we expected this work to cost about $10,000. And now that the pipeline is built and verified, we expect the audit budget for the next state to be close to zero. The most amazing thing is that our agent 4A, running on Astra and then later on Opus, found most of these approaches incrementally on its own. By building the system, running it, debugging it, and trying new things. It ran for 10 days, tried over a dozen architectures. Its hangs ran for more than 4,000 times alongside 90,000 calls to Jeff, tried practically every model on the Pareto frontier, and the models inside those runs went through roughly 6 billion tokens. And once all of that was done, we wrote what we learned back into our library of agentic parts. This work leaned on these existing ones and we're adding new tested ones like the checking letter. This is to me the value of taking work off the main thread for humans and for agents because every single one of these iterations was a version controlled Hank. They were easy to revert, understand, debug, and critique. So, can models do this natively now? Not quite. They're missing four things which we'll we'll go over and there's something that doesn't go away though as models get bigger and more capable, the prior. Just like humans, models bring their own priors, their beliefs about what should be in the world to every task. This is the fourth schema that I mentioned before, the one the model brings with it. Bigger models in a strange way are worse at this sometimes. They hold onto their beliefs and argue for them even more vehemently. Here's a few that we ran into. One pipeline looked for in kind spelled with an underscore. The file spelled it with a capital I and a hyphen. So, the model counted 15,658 in kind contributions as ordinary money, didn't double-check. The loan payment repayments also disappeared the same way. Another one tried four date formats when the file used the fifth. 99.4% of transaction dates then came out empty and every row count and dollar so matched so the agent moved on. Several runs noticed that loan amounts were negative, decided that the sign was redundant, and stored the absolute value. A net of 784,000 became 5.3 million dollars of debt in the end, validated by multiple models. And this is quickly becoming the largest class of error I see across companies at data work as models take over. Sometimes you can solve this just by not vibing, by checking results and applying human intuition. And what you're actually doing there when you do this is interesting. You're intersecting your priors with the models. In that first example, any human would catch missing in-kind contributions if they knew campaign finance or thing to check. So, we found it. But humans, speaking as a human on this particular task, are expensive. They are slow, they pause timelines, they're even more non-deterministic than models, and they hate interruptions. And what it turns out you can also get really far with cross-model prior intersection. On our 200 held out families, Jev and Luna agreed on 194. And none of those shared answers was wrong. Every error sat in the six families where they disagree. Models from different families, different pedigrees, tend to disagree more and have different priors than models in the same group, for example. The exception, I'm not going to say who, are models that are distilled outside the same company. So, with that kind of work, you can shrink the surface quite a bit, and the errors along with it down to a much smaller but persistent class of problems that apply to LLMs as a species. So, in one benchmark, four different model families produced the same total down to this end because they all shared a sign convention that the answer key didn't. And once you get that far, the only way forward is to list them, look out for them, and apply specific solutions. So, going back, what are the four things we're missing to be able to self-optimize? Number one is a stable deployment surface. We use Hank Weave for this, a harness that takes the fast-moving surface of agent runtimes like cloud code and pie, and then exposes a consistent, well-tested log and API interface with checkpointing, rollback, budget enforcement, everything built in. Something you can build on. But other things fit this description. To be fair, for headless data work, Hank Weave, I would say, still the oldest, most stable, and the most open. I'll leave the skill up if you'd like to try it. Number two is some way to bound your experiments. You can't test unless you can control cost, time, and access as resources. The third one is know-how and methodology. We found that this new class of models can take everything we've learned from two years of deployment, agentic patterns, debugging techniques, prompting, and apply to complex problems. With the right abstraction layer, you can express it very token efficiently, and the models can implement it. The fourth and the most important one is a failure surface. Archimedes said, "Give me a place to stand, and I will move the Earth." And in our case, we haven't met a problem or a constraint that we couldn't iterate ourselves to solving given a stable stack to build on and a failure surface to iterate against. That last one has been the hardest to put into words and into tooling and instructions that helps models get there. The current best models are still only barely good at this, sparks of intelligence, so to speak. The whole process though involves going through six stages. First, we inspect the job and we split it. For Jeff, that means splitting some 800,000 rows into 200,000 smaller, specialized cases, and each of those cases into typed questions. What kind of donor, one donor, you know, different different questions that needed to get answered. Then we decide the failure surface. In this case, we ran good models on a small slice of the data with some of it being manually checked to set a baseline that we can fail against. In other cases, we've set automated failure systems, we've set schema checks, we've also covered behavior as a thing that we can measure against, but it's very important to have to define a failure surface before you ever start. But now comes the fun part. We want minimal bounded pilots to test architecture. And once something starts working, we move those tags into rigs. Code that runs between bounded work segments that we call codons. These codons are then reused, looped independently, validated, and placed in sequence. Finally, working pipelines get deployed to production, where they can slowly mature. And that middle part is where the surfing happens. We find an architecture that gives a solid result, we use it as a baseline, and we swap models and some other parts for faster and cheaper options, compare against known intermediates, tighten the requirement for success, each time moving closer and closer to the ideal, and you just watch the frontier move. Being able to break a complex problem into pieces means each piece can move its own to its own best position, as Adam Smith would say. In a system like this, ontologies exist as intermediates between intelligence and result, which makes them easier to reuse. That's the other half of what I mean by just-in-time ontologies. It's built when the job needs it, kept with its receipts, and reused when the next job needs the same thing. And once you have those, you can make these things cheaper. You can you can run them in parallel across data sets to scale up. I close with an apology, right? That the answer is there is no ontology. Not a universal one, at least. There never was one. The good news is that this makes every ontology and schema valid, as long as you can label and preserve the context around it. And if we can harness this cheap intelligence is starting to percolate through the world around us, we're rapidly approaching a future where any information, in any format, is made as useful as it can be. And at Outpace, we're working as hard as we can to make that future a reality. Thank you.