Alex Shaw and Ryan Marten (Laude Institute) present a rollout-centered approach to evaluating and improving AI agents. Drawing on their work on Harbor, Terminal-Bench, and OpenThoughts-Agent, the talk connects sandboxed environments, agent evaluations, and optimization workflows into a practical framework for generating and learning from rollouts. # Channel Identity Observations: Terminal-Bench / Harbor Talk (Alex Shaw) ## Explicit Self-Description - **Speaker affiliation**: "my name is Alex Shaw. I work at Lot Institute" (transcribed name for Laude Institute). - **Product scope claim**: Harbor is described as "an agent evaluation and RL environment framework." - **Three-part definition of Harbor**: 1. "a format for specifying agentic environments" 2. "an open-source framework for performing rollouts in parallel using any agent with any model in any sandbox on any task" 3. "a registry of popular training and aval [eval] environment sets" - **Positioning claim**: Harbor is "trying to be a common language for environments" that is "extremely interoperable." ## Claimed Scope and Breadth - **Ecosystem scale claims**: "I think we have like three or 400 Aval sets by now. And in fact, I think two or three benchmarks even came out today that run with Harbor." - **Named adopters** (creator-asserted, not independently verified): Cognition ("Swix saying that his team at Cognition migrated all their valves to harbor"), Poolside ("uses Harbor to do all of their evaluations for model training"), Scale ("launched their whole Sweet Atlas suite on Harbor"), LangChain, Snorkel, Tinker, Ramp (RampBench). - **Adjacent market claim**: "there's actually like a multibillion dollar market right now that exists probably around harbor data." ## Intended Audience - **Developers and companies building agents**: "the answer is every single company that uses computers" should be building evals. - **Practitioners at labs and enterprises**: examples reference labs training models, SFT/RL pipelines, and internal codebase benchmarks. - **Recruiting audience**: "we're actually actively trying to grow the Harbor team right now. So if you want to get involved then shoot me a DM on Twitter." ## Observable Presentation Patterns (Demonstrated, Not Claimed) - **Technical, demo-driven format**: live CLI walkthrough ("harbor run... terminal bench 2.1 with codeex and GPT 5.5... run 64 in parallel") and a `harbor exec` demo with map/reduce prompts. - **Trend-thesis framing**: opens with a "2018 vs. now" narrative — "software engineering was when you knew what the code would do before you ran it" — generalizing François Chollet's tweet (transcribed as "France Chalet") to "agents in general are a form of machine learning." - **Use-case taxonomy**: four eval categories people build — how well agents *build* products, *use* products, *power* product features, and *automate internal processes*. - **External validation via quotes**: cites a Satya Nadella quote — "if you want to build an agentic system start with the aval that matters... I welcome all models." ## Claims vs. Demonstrations — Distinctions | Type | Evidence | |---|---| | **Creator claim** | Ecosystem adoption figures, benchmark counts, multibillion-dollar data market | | **Demonstrated** | Actual CLI usage, rollout pipeline mechanics, exec/map-reduce workflow shown on screen | | **Third-party claim relayed** | Nadella quote, Chollet tweet, adopter testimonials | No private motivations beyond stated recruiting intent were inferred; adoption and market-size figures remain creator-asserted and unverified within this material. # Delivery Observation: "Everything Is a Rollout" — Alex Shaw ## Tone - **Conversational and casual**: frequent filler words — "Um", "Uh", "so yeah, like you said". - **Light and playful**: opens with a nostalgic 2018 time-travel bit (Avengers, TikTok rebrand, AirPods) before the technical content. - **Enthusiastic about the ecosystem**: rapid-fire listing of community benchmarks near the end, "this is one of my favorites." ## Vocabulary & Framing - Consistent **ML↔agent-development analogy vocabulary**: training data, validation set, loss function, backprop, overfitting mapped to environments, evals, rewards, pull requests, reward hacking. - Signature reframing move — explicitly generalizing others' claims: > "He says agentic coding and my claim is well just agents in general are a form of machine learning." > "generated code is best treated as a blackbox artifact and I say agent performance itself is best treated as a blackbox artifact." - Repeated superlatives/universals: "every single company that uses computers", "extremely universal", "any agent with any model in any sandbox on any task". ## Teaching Habits - **Anchoring on external quotes/tweets** (twice: the 2018 code-review tweet, François Chalet's tweet) and building the argument from them. - **Contrast structure**: 2018 regex program vs. 2026 model-call version of the same phone-extraction task; "left/right" ML vs. agent-building table. - **Numbered enumeration**: "what is Harbor more specifically? One... Two..." and "I'm going to list four things" for eval types. - **Concrete examples over abstractions**: RampBench, HubSpot, Stripe, Runescape bench, receipts, Obsidian files. - **Live demo narration** with self-commentary: "let me see if I can pause this", "Oh, shoot", "But uh yeah, you can kind of see here". - **Signposting transitions**: "and that brings us to our next question", "so who needs a val?", "in the last couple minutes I just want to talk about...". ## Recurring Phrases - **"Rollout"** as the organizing concept, per the title — appears throughout ("Harbor rollout", "prod rollouts", "10,000 rollouts"). - **"So"** as a near-constant sentence starter and connective. - **"Kind of" / "sort of" / "like"** as hedging qualifiers throughout. - "That's your verifier", "that's your val", "that becomes your val result" — **definitional "that's your X" pattern** when introducing components. - "Which is useful because...", "which means you can..." — trailing justifications. ## Communication Patterns - **Self-aware simplification caveats**: "this process here which looks relatively simple is actually... a little bit overly simplified." - **Honest asides about limitations**: "rollouts can take a while", "It's going to go away kind of soon" (re: Harbor Exec). - **Personal-practice examples**: uses his own Codex sessions and tool choices in the map-reduce demo ("cursor CLI because it's cheap and fast... fable 5 because I... care about intelligence"). - **Action-oriented close**: hiring pitch — "if you want to get involved then shoot me a DM on Twitter." - Occasional restarts/stutters ("the the the bare bones approach", "what what harbor does") consistent with live, unscripted delivery. ## Uncertainty Notes - Filler density and stutters suggest live improvised delivery, but this cannot be fully confirmed from transcript alone (video playback is only referenced, not observable). - Some apparent transcript spellings ("rejax/reax", "aval", "Fable 5", "France Chalet") are likely transcription artifacts; quoted them as transcribed rather than as delivery features. # Position Record: "Everything Is a Rollout" — Alex Shaw **Speaker:** Alex Shaw, Laude Institute (self-identified in transcript) ## Core Thesis - **Agent development is machine learning, not software engineering.** Shaw argues agent performance should be treated as a black-box artifact managed via empirical evaluation, generalizing François Chalet's tweet about agentic coding. - Central analogy framing the talk: *"Everything is a rollout"* — the simple loop of task → sandbox → agent → verifier → reward underlies all Harbor use cases. ## Key Claims and Reasoning ### 1. The paradigm shift from 2018 software engineering - Claims 2018-style software engineering is defined by: *"software engineering was when you knew what the code would do before you ran it."* - Contrast: a regex-based phone-number extractor is deterministic over a million runs; replacing the regex with a model call yields a more powerful but non-deterministic program, and uncertainty grows with task complexity. - Quotes François Chalet: agentic coding output should be treated as a black-box artifact managed empirically; Shaw generalizes "generated code" to "agent performance." ### 2. ML ↔ agent-development analogy Shaw maps ML concepts to agent-building equivalents: - Training data → **environments** - Test/validation sets → **evals** (which "are actually just environments") - Weights → **skills, prompts, tools, model choice** - Loss function → **environment rewards/feedback** - Optimizer → **text-based context optimization** (e.g., "Geepa"/Jeefa-style loops, running a coding agent in a loop) - Gradient step → **pull request** - Overfitting → **reward hacking** ### 3. What Harbor is - A **format** for specifying agentic environments (file-directory layout, now "relatively standard" and interoperable). - An **open-source framework** for parallel rollouts with any agent, model, sandbox, task. - A **registry** of eval sets ("three or 400 by now"; "two or three benchmarks came out today"). - Positions it as a common language for environments maximizing "data velocity." ### 4. Who needs evals - Claim: **every company that uses computers** should build evals, since AI automation of computer processes increases productivity and value. - Quotes Satya Nadella (Applied Compute podcast): start with the eval that matters and the ability to grade outcomes — "I welcome all models." Shaw's reading: owning an eval frees you from trusting brands or others' evals and lets you balance cost/performance. ### 5. Four eval categories observed in practice 1. How well agents **build** your product (e.g., Ramp's "Rampbench" on internal codebases). 2. How well agents **use** your product (headless mode trend; HubSpot, Stripe cited as developer-first). 3. How well agents **power product features**. 4. How well agents **automate internal processes**. ### 6. Rollout use cases beyond evals - **Task/data market:** multibillion-dollar market where experts sell task specifications to labs. - **Prod rollouts / "agentic map-reduce":** mass parallel agents with aggregation — reward-hacking detection, receipt processing, Obsidian search, PR analysis. Noted as **emergent** (not originally built for); motivated the Harbor Exec feature (temporary, "going to go away soon"). - **SFT on trajectories** and **RL on rewards + trajectories** (Tinker integration cited). - **Other optimization:** evolutionary methods, auto hill-climbing on harness/scaffold. ### 7. Ecosystem claims (asserted, not demonstrated) - Users/adapters cited: Frontier Suite, Banker Toolbench (Handshake), Cognition (migrated evals), RuneBench (Runescape benchmark), Scale's Sweet Atlas, Poolside, Auto Agent (Kevin Goo), Aperture Query post-training, Cognition's Frontier Code, LangChain deep agents, Snorkel's Senior Sweet Bench (released "today"). ## Qualifications and Tensions Preserved - Shaw admits the basic rollout description is "overly simplified" and Harbor supports many variants. - Rollout durations can be long, hence the push for parallelism — throughput is a stated constraint, not solved. - Evals are environments "at least the style that we'll be talking about today" — a hedge on generality. - Harbor Exec is explicitly temporary. - Closing includes a **recruiting pitch** (team growth, Twitter DMs) — a stated motivation for the link choice, alongside docs. ## Unverified Items - Third-party claims (Rampbench details, Nadella quote wording, benchmark counts) are recorded as stated by Shaw; no independent verification in source. Speaker identifications of ecosystem projects rely solely on Shaw's statements. # Speaker Attribution & Provenance Record ## Video - **Title:** Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute - **Video ID:** `01a15ca8-3e4c-7347-901e-72e387862304` ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Alex Shaw** | **Creator / Primary presenter** — introduces himself, "I work at Lot Institute" (transcribed; likely "Laude Institute" per title), presents talk on Harbor | Self-identification at talk opening | - **No other speakers** are identified in the transcript. The title references **Ryan Marten**, but he does not speak in this transcript segment. The "\>\>" at the end suggests a possible second speaker not captured here. - No **narrator**, **guest**, or **host voice** appears beyond the brief emcee-like framing ("like you said, my name is Alex Shaw"). ## Referenced Third-Party Claims & Quotes - **Tweet quote:** "Stay Sassy Sassy" (X/Twitter user) — humorous tweet about reviewing a 400-line pull request in 2018. - **Tweet quote:** "France Chalet" (transcribed; likely **François Chollet**) — "Agentic coding is a form of machine learning…" (quoted twice, paraphrased/extended by the speaker). - **Quote attributed to "Satcha Nadella"** (transcribed; likely Satya Nadella), from the "applied compute podcast he did last week" — "if you want to build an agentic system start with the eval that matters…" (recency claim is speaker's, unverified). ## Referenced Third-Party Products / Organizations - **Products/platforms:** Harbor (speaker's org), Terminal-Bench 2.1, RampBench (Ramp), Frontier Suite, Banker Toolbench (Handshake), RuneBench (RuneScape), Scale's Sweet Atlas suite, Tinker, auto-agent (Kevin Goo), Frontier Code (Cognition), LangChain deep agents, Senior Suite Bench (Snorkel), Cursor CLI, Modal, GPT-5.5, Codex, Fable 5, Jeiba (transcribed; likely "GEPA"), Databricks, TikTok, HubSpot, Stripe. - **Claims by others:** Swix (X post) — Cognition migrated their evals to Harbor; Cognition and Poolside cited as Harbor users (speaker's claims). ## Promotional Segments (Flagged) - **Harbor promotion** throughout — the talk is a presentation of the speaker's own product ("we built Harbor"). - **Explicit recruitment/promo closing:** speaker links his Twitter, asks viewers to DM him to join the Harbor team ("we're actively trying to grow the Harbor team"). - **Feature demos:** `launch` command rollout ("something we're rolling out soon"), Harbor Exec ("going to go away kind of soon") — product announcements. ## Availability & Metadata Limits - **No curated metadata supplied** beyond the video title. - **Comment availability:** not indicated. - **Transcription uncertainty:** several names appear phonetically transcribed ("Lot Institute," "France Chalet," "Satcha Nadella," "Jeiba," "fable 5"); identity inferences noted above remain **explicitly uncertain**. - **Embedded videos:** the speaker shows a demo video ("I actually have a video") of Harbor runs; contents described only by the speaker, not independently transcribed. Awesome. \[music\] Thank you so much. Uh so yeah, like you said, my name is Alex Shaw. I work at Lot Institute. Um, and I'll be speaking today about Harbor, which is an agent evaluation and uh, RL environment framework. And the title of my talk is everything is a rollout. And I think you'll see as I get into it why we titled the talk that way. Um, but first I want everybody to come travel back in time with me to the year 2018. So eight eight years ago. Um, and we're going to talk about some of the things that were going on in 2018. So, uh, probably you're you're going to see Avengers Infinity War later today. The second one is just getting released. Uh, GPT1 was was just released and you probably didn't even notice, although maybe some people did. Um, musically, this random startup was about to rebrand to a product called Tik Tok. And you might have just learned about AirPods when you saw somebody walking around with headphones that had no cord. Um, so what about what about software engineering? What did software engineering look like in 2018? Uh, we're going to read this tweet from Stay Sassy Sassy with two two A's. Um, about what it was like to write code in 2018. So it says, "It's 2018 and your coworker just sent you a 400 line pull request. You get a cup of coffee and sit down to review it. It's beautiful. Elegant micro refactors, crispy method names. You catch a few things, but that's okay. It's part of the dance. They didn't consider extensibility on part of their API. Here's a comment, buddy. And this is actually just part of the tweet, so it keeps going. You should look it up if you want. And it's obviously written humorously. Um, but the thing is, it does feel a little bit nostalgic. Um, just like some of these other things that were going on in 2018. Now, the thing is, uh, this only changed maybe six or 12 or 18 if you're a very early adopter months ago. Um, but it but it already feels like the distant past in some ways. Uh, and I think it's time to start talking about well, what will the history book say about software engineering? And when I say software engineer, I mean that style of 2018 software engineering. Um, it will probably say the the the history books will probably say a lot of things. Um, but I think one thing for sure that they'll say is software engineering was when you knew what the code would do before you ran it. Um, so and that brings me then to a different tweet from France Chalet uh where he says, "Agentic coding is a form of machine learning. Generated code is best treated as a blackbox artifact whose behavior and generalization should be managed via empirical evaluation like with any ML model." Um, and that kind of brings us to the next part in this talk, which is to compare and contrast what agent development looks like versus what more traditional software engineering development looks like and why it demands a new set of tools to really understand what's going on and have confidence and trust. Um, so here's a here's a 2018 program right here. Uh, so you can tell already the purpose of the program is to extract phone numbers from text. And we have a rejax right here that uh looks for the phone number. And I can say with 100% confidence what will happen if I run this program 1 million times in a row. Um so now let's update it to the 2026 version. So uh I swap out my reax and instead obviously I throw in my model call instead and I say extract this phone number. Um, so in some ways this is actually a more powerful program because uh the reax was actually a little bit brittle. It would have missed any phone number that wasn't formatted exactly like how it was specified. Whereas I'm pretty confident that this program with GPT 5.5 will catch a lot of the phone numbers that are formatted weirdly. Um, however, if I ran this exact program one million times, I'm not 100% confident that it will print the same thing every single time or that I know exactly what it will print. Um, it probably gets it right almost every time. This is a pretty simple task. Um, but, uh, this is obviously far simpler than the things we're asking these models to do. And the uncertainty only increases as the complexity of the task increases. Uh so now let's come back to Francois's tweet and let's update it a little bit. We'll generalize it. So he says agentic coding and my claim is well just agents in general are a form of machine learning. And then he says generated code is best treated as a blackbox artifact and I say agent performance itself is best treated as a blackbox artifact. Uh and that brings us to our to our new paradigm. So uh we know now agent development is more similar to machine learning than it is to software engineering. So what are the things that we should keep an eye out for the the tools that you use for machine learning and also the pitfalls of machine learning and what are the analoges for those with agent development. So we have machine learning on the left agent building on the right. Uh so training data that now looks like environments and then your test and your validation set that looks like a val which if you take off the mask is also actually just environments at least the style that we'll be talking about today. Um weights your model weights are now skills prompts tools the model whether you're picking between models or actually updating the model yourself. uh your loss function looks now more like environment rewards and feedback. Your backrop or optimizer is now some context textbased optimization algorithm like Jeepa or even just running a coding agent in a loop. Uh your gradient descent step looks like a pull request into your repo. And overfitting looks like reward hacking or also just uh overfitting that's also possible for agent development. Um, so we have these things on the left, there's, you know, products and libraries and platforms that were built for these purposes. And then for everything on the right, we're just getting started. So, uh, we built Harbor to answer some of these questions. We're building it to answer more of these questions, but other people are also building interesting products and interesting frameworks to help people tighten this loop of agent development. Um and then something else to consider is that uh as popular as machine learning was and it spawned some of the largest companies in the world, you have like data bricks which are worth uh hundreds of billions of dollars. Um already the number of people using and building agents is probably magnitudes larger than the number of people that ever uh were doing machine learning and that trend will will only continue. Uh and now let's look at the last piece of Francois's tweet that I want to call out uh which is uh these these agent agent performance uh dut should be managed via empirical evaluation. Uh and that brings us to our next question which is how do I actually evaluate an agent? Uh and this is when we start to get into uh what what harbor does. So the answer as we already gave away earlier is environments. That's your aval and your training data. And what is an environment in this case? Well, we need an instruction. We need some way to tell the agent what it's supposed to do. And then we need some place for the agent to try to do this thing. And right now all of the agents run on computers and they do stuff on computers. So we'll put it into a computer, but we'll put it into a virtual computer. So a sandbox. Um, and then we need some way of telling whether or not the agent actually did the thing that we told it to do in the sandbox within some amount of time or other stopping condition. And that's your verifier, which is like some programmatic tests or rubrics um or agent that comes in to see what happened. So in Harbor, you specify this as a file directory. And this uh specific directory layout has become relatively standard in a lot of the environment space. So lots of people have adopted it and use it as a way to specify environments which is useful because then it can easily pass between hands and becomes interoperable. Um and then okay cool we have a bunch of data now we've implemented a bunch of these environments. Now how do I actually use it to start understanding what my agent can and can't do and how to make it better. Um so that's a harbor roll out and uh so what you do is you start with your tasks and then you take it you start up your sandbox and then step one you pass that sandbox to your agent. So you're either running your agent outside the sandbox and executing commands into it or you're running it inside the sandbox and it's calling whatever its commands are as part of its program. Uh, and then it runs for some amount of time until it hits a stopping condition, produces some trajectory, and we'll come back to that later because that's important. And then you pass that sandbox to the verifier, which uh runs some verification process. And then finally, you stop the sandbox. The verifier produces a reward or a set of rewards and then you take those, you aggregate them a bunch across a bunch of rollouts in a data set with some agent, and that becomes your valid result. So this process here which looks relatively simple is actually extremely universal and uh it's also a little bit over overly simplified. So Harbor by now allows for a lot of different flavors of rollout. So you can do multi-step, you can run your verification in a separate sandbox, you can collect artifacts, you can simulate a user. Um but this is kind of like the the the bare bones approach uh that they're all built off of. So what is Harbor more specifically? One, it's a format for specifying agentic environments. Two, it's an open-source framework for performing rollouts in parallel using any agent with any model in any sandbox on any task. And it's a registry of popular training and a val environment sets. I think we have like three or 400 Aval sets by now. And in fact, I think two or three benchmarks even came out today that run with Harbor. Um, so we're excited about that. And in general, it is trying to be a common language for environments. So a way for people to specify things that are extremely interoperable and it allows you to maximize data velocity and just increases progress in the industry. Uh so who needs a val? Uh now that we understand how to make them, we understand how to use them. Well, now who should actually be doing this? And the answer is every single company that uses computers. And I think that's probably close to all of the companies in the world. And the reason is because if you're doing something on a computer, then you should be seeing if you can use AI to automate part or all of that process that you're currently performing on a computer. Uh because that will increase the productivity of your company and therefore increase the value that your company generates. Uh so I like this quote from Satcha Nadella from the applied compute podcast he did last week. He says if you want to build an agentic system start with the aval that matters and your ability to grade the outcome and then say I welcome all models. So I like that last line because what he goes to say is that as soon as you have an aval the power is now in your hands. You can consider every single model. You don't have to trust brand. You don't have to trust somebody else's aval. You don't have to trust a public aval. Um and you can kind of skate the paro however you desire to uh balance that cost performance tradeoff. So step one build the eval step two kind of optimize against it. Um so what will people evaluate and uh I'm going to list four things and these four things are based off of what we actually see people using harbor to build a val for right now. Uh so so the first type of a val is people evaluating how well agents build their products. So everybody that's building a software product has some internal codebase or set of internal code bases. Um I think ramp recently announced like rampbench which is aubench built off of their internal codebase and what that allows them to do is pick the coding agent or the model that performs best on their internal use cases and they don't have to you know maximize token spend. instead they can make informed and educated decisions about how they build their products with agents. The second type of AVL that we see people build is how to evaluate uh how well agents use your product. So anybody that builds a software product is probably moving towards a world where they offer some sort of headless mode. So you see some companies have obvious have always been this way. They've been developer first like uh HubSpot and Stripe and things like that. And the idea is uh if you can make an aval to see how well agents use your product and then iterate on your product to make it more usable for agents, you're going to get more usage and therefore your product uh will become more valuable to agents. And then uh three evaluate how well agents power product features and then four evaluate how well agents automate internal processes. So depending on what type of company company you are, uh one or more of these might apply to you, but everybody should be considering right now how they can do one of these things. Um okay, so what are the different use cases of the harbor rollout? So we've talked about aval that's the most popular right now. Um oh oh here we go. I actually have a video even of us doing an eval in harbor. So this is uh how you can run it from the command line or have your agent run it from the command line. So you can see here it says harbor run. We're running in this case terminal bench 2.1 with codeex and GPT 5.5. And we're saying let's run 64 in parallel. And then we actually type in the launch command right here which is something we're rolling out soon which allows you to launch the rollouts on harbor servers which means you can just fire and forget and go to sleep and then come back the next day and 10,000 rollouts are done for you. And you can kind of see in here what it looks like to look at those rollouts happening. Um, but they don't complete in the time span of this video because rollouts can take a while. Uh, which is another case for parallelizing as much as you possibly can to tighten that uh that loop and maximize your throughput. Um, so there are people who create tasks and sell those as data. So there's actually like a multibillion dollar market right now that exists probably around harbor data and also other types of data. Um but it's uh very uh I guess lucrative right now where experts can specify certain types of tasks that they would like automated and then sell those to one of the labs that's training models to uh improve on that capability. Uh you can also do what we call prod rollouts. So remember harbor is evaluate any agent with any model. Uh I guess not even evaluate we'll just say uh run any agent with any model in any sandbox on any task. And that actually doesn't mean you have to do it for evaluation. It also doesn't mean you have to do it for training. It could literally be that you want to do what we've been calling agentic map reduce which is you just want to run a ton of agents on distributed compute. of sandboxes in parallel and then somehow aggregate those results probably also intelligently. So things like looking over a bunch of trajectories to detect reward hacking or I don't know like processing a bunch of receipts for reimbursements or searching over your obsidian files to figure out uh semantically what where you wrote some note or processing a bunch of PRs to ask a question about it. So we see a bunch of different use cases. This is actually an emergent use case. So, we didn't build Harbor for this, but we see people doing it a lot. So, we actually built this feature and launched it just for this. So, it's Harbor Exec. Uh, it's going to go away kind of soon, but you can see in this scenario, actually, let me see if I can pause this. So, uh, I'm actually going to go back a little bit before we kicked it off. So, I'm saying Harbor Exec input is all of my codec sessions from the June 20s. And then I say prompt if I corrected the agent write an analysis.json with mistake reason and correction. And then my reduce prompt is summarize recurring mistakes and failure categories into a concise feedback.mmd file. And you can see in this case I'm running on modal. I do the map step with cursor CLI because it's cheap and fast. And then I do the reduce step with fable 5 because I wanted to have like an accurate summary and I care about intelligence in that scenario. So, and then I'm limiting it to 32 because I didn't actually want to process all of my sessions for this demo. But you could there's no reason you couldn't do like 10,000 sessions or a thousand sessions and then maybe you have to go and understand the results more deeply. But um Oh, shoot. But uh yeah, you can kind of see here I think. So we kick this off. A bunch of these rollouts start in parallel. This one is running locally on my computer with like cloud sandboxing but orchestrated locally. And then you can see here all of the cursor rollouts running right now. Uh but those will take a second to finish. And then you can see the reduce step here where Fable recommended me like recurring mistakes. And like these recurring mistakes, for example, could be used to inform the next batch of Harbor tasks that I create to evaluate my agent and pick like a better agent or train up a scale or something like that. So uh another use case of Harbor. And then we see people taking the trajectories and doing SFT. And then we also see people taking the reward or rewards the trajectory in the form of tokens and doing actual reinforcement learning. Uh for example, Tinker launched an integration with Harbor. And then uh people also do other types of optimization. So we mentioned JEA but any of these like evolutionary methods you take the text feedback in the form of a trajectory and the Aval and you can do some sort of like auto hill climbing with a harness or a scale. Um so in the last couple minutes I just want to talk about some cool things people have built with Harbor. I'll try to breeze through this. Uh so uh Frontier Suite and ultra long horizon software engineering benchmark built on Harbor Banker toolbench and investment banking benchmark built by handshake on Harbor. Um Swix saying that his team at Cognition migrated all their valves to harbor. This Rune bench this is one of my favorites. It's like a benchmark to measure how well agents can play Runescape. Um Scale launched their whole Sweet Atlas suite on Harbor. Uh, pool side uses Harbor to do all of their evaluations for model training. Uh, Kevin Goo created auto agent which is a self-optimizing agent that all you have to plug in is a Harboral. Uh, after query post train a model using Harbor Cognition released Frontier Code recently which is a Harbor benchmark and lang chain just integrated deep agents and their sandboxes into Harbor. Uh and then actually just today, Snorkel released Senior Sweet Bench, which is a benchmark for measuring how well agents can function under ambiguity with behavioral feedback. So, uh that's kind of a brief overview of Harbor and the different things you can do with it. I would encourage everybody here to check it out. And then this is a link to my Twitter. And the reason I'm linking my Twitter, usually I link Harbor Docs, but we're actually actively trying to grow the Harbor team right now. So if you want to get involved then shoot me a DM on Twitter and if you just want to see the docs then click the docs link on my Twitter and that's the best way to get there. Uh so thank you very much \>\> \[music\]