Suraj Gupta, who leads harness development at Warp, presents three practical mechanisms for making a software factory improve over time, with demos from Warp's open-source factory: (1) self-improving skills, where an outer-loop agent reviews a triage agent's runs and opens pull requests to update its skill, keeping changes Git-tracked and human-reviewed; (2) persistent, versioned memory so agents like a Sentry agent can reuse previously discovered root causes, working across harnesses including Claude Code and Codex; and (3) model routing to avoid paying premium model prices for simple tasks like triage or CI fixes, using Warp's auto models or custom rules. Warp's internal evals reportedly found UI tasks run well on GLM. # Video Identity Observations — "Building Self-Improving Agent Software Factories" (Suraj Gupta, Warp) ## Speaker & Channel Context - **Speaker self-identification**: "my name is Siraj Gupta. I lead Harness development at Warp." - **Company origin story, as stated**: "we started off by building a state-of-the-art agentic development environment... born out of a modern-day terminal." - **Current positioning, as stated**: "nowadays we're building a... cloud agent platform to enable teams to build software factories." - **Claimed scale**: "about a million active users that use that." ## Intended Audience Evidence points to a **professional developer / engineering-organization audience**, not hobbyists: - The talk is framed as practical takeaway for practitioners: "hopefully you can walk out of here with something concrete that you can implement yourselves." - Audience familiarity is assumed throughout: "You've probably heard of both of these terms, self-improvement and software factories, used throughout this conference." - Work-centered framing: "this is going to become an extremely important part of our jobs as software engineers as we transition from building products to building factories." - Cost-pain appeals to team leads: "it can become prohibitively expensive to run all of your agents... I'm sure some of you have maybe been burnt by this." ## Claimed Scope of the Talk - Focuses on **three concrete self-improvement mechanisms**: "I'm going to focus on three concrete ways that agents can improve over time... skills... persistent memory, and then... model routing." - Explicitly narrows a broad topic: "self-improvement is a pretty broad umbrella, but..." - Claims novelty in the intersection: "less has been said about how software factories themselves can self-improve over time." - Acknowledges buzzword risk: "which I know might sound kind of buzzy." ## Product / Brand Evidence (Creator Claims) - **Warp** terminal and agentic dev environment; **"Oz"** cloud agent platform where "you can attach memory stores" to agents. - **Internal dogfooding claim**: "we've open-sourced our client repository and it's fully run as a factory." - **Product features demonstrated**: outer-loop agents updating skills via pull requests ("a human is actually going to review those updates"), versioned memory stores, and "auto models" model routing with custom rules ("I want database migrations done with... GLM"). - **Roadmap claims**: building customer-facing evals and routing into the product so "our customers can actually use it in their own workflows." - Mentions ecosystem compatibility: "works across all harnesses... Warp Zone, proprietary harness, plot code, Kodak" (transcript spellings uncertain — likely Claude Code and Codex). ## Observable Presentation Patterns - **Demonstration-heavy style**: live walkthroughs of GitHub workflow configs, memory store UIs, and Warp desktop model routing configuration files. - **Concrete example anchoring**: triage agent on GitHub issues and Sentry issue-fixing used as recurring worked examples. - **Candid engineering tone**: admits uncertainty ("At this point though, that's kind of more of an art than a science"). - **Human-in-the-loop emphasis**: repeatedly stresses human review of agent-produced skill updates and editable, traceable memories. - **Conference context**: closing invitation — "we're around in a booth... come talk to me" — indicating in-person developer-conference distribution rather than standalone channel content. ## Distinguishing Claims vs. Demonstrations - **Claims**: 1M active users; internal factory productivity; planned customer-facing evals. - **Demonstrated in-transcript**: skill-improvement outer-loop via PRs, memory store UI with source traceability, and custom model routing rules — shown via described screen demos, not independently verified results. # Delivery Observation — Suraj Gupta, "Building Self-Improving Agent Software Factories" ## Overall Communication Style The delivery is **conversational, demo-heavy, and incremental**. The speaker moves between slides and live product screenshots, frequently narrating transitions ("Um so, let me swap here. Cool.") and re-orienting the audience after each demo. ## Recurring Verbal Habits - **Heavy filler usage**, especially "um" at clause starts: *"Um my name is Siraj Gupta. Um in case you haven't heard of Warp before, um we started off by..."* - **"So" as a pivot/resumption marker**, appearing dozens of times: > "So, how do we fix this and let the agent improve over time using that skill?" > "Um cool. So, skills are great for remembering procedures, but what about everything else?" - **"What's cool / What's really nice" as a highlight device** before key points: > "What's really nice about this is that um the So, in step four, you'll see we we added the triage skill..." > "What's cool in Oz is that persistent memory works across all harnesses" - **"The idea is that..."** as an explanation opener, used repeatedly: > "The idea is that your um your inner loop agent is the thing that's actually applying the skill." > "The idea is that um you you get an issue that comes in on GitHub..." ## Teaching Patterns - **Roadmapping then fulfilling it**: he previews the structure explicitly — *"I'm going to start with um skills, then we'll talk about persistent memory, and then I'll finish off by talking about model routing"* — then closes each section with a signpost ("so that's that's kind of what what I want to talk about for skills. Um let me talk about persistent memory now"). - **Concrete anchor examples per concept**: triage agents for skills, a Sentry/root-cause agent for memory, cost-of-Opus for routing. - **Anticipating audience questions**: *"So, um you may be wondering how this plays into self-improvement loops for um factories."* - **Product demos interleaved with slides**, including candid troubleshooting on stage: *"Got to drag this over here."* / *"Um, let me go back to the deck. Give me a sec."* ## Vocabulary & Framing Choices - Terms repeated as a lexicon: **inner loop / outer loop agent**, **skill**, **memory store**, **harness**, **software factory**, **triage**. - Informal register even for technical material: *"I'm always down to talk about this kind of stuff"*, *"or have heard stories of of or organizations being burnt by this."* - Self-aware framing of buzzwords: *"which I know might sound kind of buzzy, um but I'm going to break it down."* ## Notable Isolated Moments (Low Confidence) - A candid mid-demo aside about a failed local setup: *"I think this my IP address here keeps changing and this my our staging platform is gated to an IP address."* — appears once; may be habitual transparency or a one-off. - Closing repetition pattern *"But yeah, thanks for your time today. That's all I had."* — single occurrence. ## Stable vs. Uncertain - **Stable**: "um"/"so" fillers, "the idea is that", section signposting, demo-with-narration structure — each recurs multiple times. - **Uncertain**: claims about how he handles mistakes or questions in Q&A cannot be assessed; only one example of live troubleshooting is visible, so no reliable pattern is claimed there. # Observation Record — "Building Self-Improving Agent Software Factories" (Suraj Gupta, Warp) ## Speaker Identification - **Suraj Gupta**, self-identified as leading **Harness development at Warp** (transcript header spells "Siraj"; title metadata says "Suraj"). No other speakers; only music follows the talk. ## Framing Claims - Warp began with an **agentic development environment** ("born out of a modern-day terminal") with about **a million active users**, and is now building a **cloud agent platform** for teams to build "software factories." - He acknowledges the phrase "self-improving software factories" "might sound kind of buzzy" and frames his goal as leaving the audience with something concrete to implement. - His stated thesis: less has been said about how **software factories themselves** can self-improve; he predicts this will become central as engineers shift "from building products to building factories." ## Position 1: Skills as Procedural Memory - **Claim:** Skills give agents procedural memory, but skills **go stale over time** due to human feedback and what agents learn in their own runs. - **Proposed mechanism — an outer loop agent:** - **Inner loop agent** applies the skill (e.g., a triage agent reproducing issues). - **Outer loop agent** observes inner loop runs, collects feedback signals (thumbs up/down, user comments, Warp employee comments), and synthesizes skill updates. - **Worked example:** Warp's open-sourced client repository runs as a factory. A triage agent triages GitHub issues; a workflow runs on every submitted issue; an outer loop agent evaluates the triage agent's decisions and updates the skill. - **Safeguards cited:** - Skill updates are opened as **pull requests**, so improvements are "tracked through git" with full observability. - A **human reviews** the updates, guarding against the outer loop agent degrading the triage agent's performance. ## Position 2: Persistent Memory - **Claim:** Skills cover procedures but not everything; without memory, agents **waste tokens re-gathering context** on previously solved issues (his example: a Sentry agent re-diagnosing a similar issue), and getting lucky on root cause is "not guaranteed." - **Proposed mechanism:** Persistent memory as a **fact store scoped to an agent**, populated by an outer loop agent extracting facts, learnings, and outcomes from inner loop runs — analogous to the skill loop. - **Worked example (Warp's "Oz" cloud platform):** - Memory stores attach to agents; a Sentry agent accumulated root-cause-analysis memories. - Memories are **versioned** and can be created, deleted, or edited by humans, though "primarily this memory creation loop is driven by agents." - Memories are **traceable to source runs**, allowing judgment calls like discarding a "local maxima" memory. (He notes an incidental demo hiccup: his IP kept changing because the staging platform was IP-gated.) - In later runs, the agent reported back which memories it used to speed up work. - **Scope claim:** In Oz, persistent memory works **across all harnesses** — "Warp Zone, proprietary harness, plot code, Kodak" (likely transcription artifacts for harness names such as Claude Code / Codex) — with memories created automatically but human-reviewable. ## Position 3: Model Routing - **Claim:** Routing tasks to models matters because running simple tasks (triage, simple CI fixes) on expensive models like Opus "can become prohibitively expensive"; he notes some organizations "have been burnt by this." - **Current Warp offerings:** - **Auto models** — out-of-the-box routers evaluated for Pareto efficiency as new models release; pitched as better for new users than pinning to Opus or Haiku. - **Custom routing rules** — configuration files mapping task classes to models (his examples: database migrations via GLM; runbooks/API documentation via Quen/Qwen). - **Qualification:** He states that choosing which models fit which tasks is currently **"more of an art than a science."** - **Work in progress:** Warp is building **customer-facing evals** so users can test, for their own workflows (not "some generic benchmark posted online"), whether e.g. Haiku beats GLM or Opus on a class of issues. - **Internal method described:** An **eval sidecar** runs a **best-at-k** approach — a prompt is run across multiple models in Oz — which led Warp to findings like "UI tasks are really well done with GLM" and don't need Opus. He notes plans to build this into the customer product. ## Overall Structure of Claimed Self-Improvement - Three mechanisms, presented in order: **skills → persistent memory → model routing**. - Recurring pattern: an **inner loop** doing the work and an **outer loop** observing and improving it, with **human review and versioning (git/PRs, memory versioning)** as stated guardrails. ## Notes on Unresolved / Tentative Items - The eval-driven model routing and customer-facing evals were described as **in progress**, not shipped ("sooner enough we're going to be building this into our product"). - The claim that task-to-model fit is "an art, not a science" remains, by his own framing, unresolved. - Some product/harness names are uncertain in the transcript ("Oz," "Zone," "plot code," "Kodak," "GLM fire," "Quen"); recorded as spoken with likely transcription artifacts. # Speaker & Provenance Record ## Video Metadata - **Title:** Building Self-Improving Agent Software Factories — Suraj Gupta, Warp - **Video ID:** `01a15ca7-298b-78bf-bbe9-a0edd46781c9` ## Speakers & Roles | Speaker | Role (self-identified) | Basis | |---|---|---| | **Suraj Gupta** (transcript renders "Siraj Gupta" once, title metadata says "Suraj") | **Creator / Presenter** — leads Harness development at Warp | Stated in opening line: *"my name is Siraj Gupta. I lead Harness development at Warp"* | | Unidentified voice | **Other** — brief non-speech audio event (`>> [music]`) at the end | Transcript tail | No additional speakers, guests, or interviewers appear. ## Speaker Content Boundaries - The **entire substantive transcript** is a single continuous conference talk by Gupta, including live product demos (Warp desktop, "Oz" cloud agent platform). - Topics covered: skills + outer-loop agents, persistent memory stores, model routing / auto models, best-at-k eval sidecar. - Closing **booth invitation** ("we're around in a booth… come talk to me") functions as an **in-person engagement prompt** — promotional in tone but not a paid or sponsored segment. ## Referenced Sources & Claims (third-party or company claims) - **Warp internal claims** (unverified in transcript): - ~1 million active users of Warp's agentic development environment. - Warp's client repository is **open-sourced** and run "as a factory." - Internal use of an eval sidecar with best-at-k model selection (e.g., "UI tasks are really well done with GLM"). - **Third-party products named:** Opus, Haiku (Anthropic models), GLM, Quen, Claude Code–adjacent harnesses ("plot code, Kodak" — likely transcription artifacts), Sentry. - No cited papers, benchmarks, or external studies are quoted; the speaker explicitly distinguishes internal findings from "some generic benchmark that someone posted online." ## Promotional Content - ⚠️ **Product promotion:** Ongoing demonstration of Warp / Oz platform features (memory stores, auto models, custom routing rules); explicit booth/engagement invitation at the end. - Speaker notes upcoming customer-facing eval features ("sooner enough we're going to be building this into our product") — forward-looking product claim. ## Availability & Evidence Channels - **Comments/transcript beyond provided text:** Not indicated; no comment-availability metadata supplied. - **Demo visuals:** Slides and screen shares are referenced verbally but not verifiable from transcript alone. - No external links, citations, or follow-up resources are stated. ## Uncertainty Flags - Spelling of speaker name differs between transcript ("Siraj") and title metadata ("Suraj") — **title metadata likely authoritative**; flag for confirmation. - Product/harness names near the end ("Warp Zone," "plot code, Kodak") appear to be **auto-transcription artifacts** of proprietary or third-party names; treat as uncertain. - All performance claims (efficiency gains, memory usage speedups, model fit) are **speaker assertions without supplied evidence**. Um my name is Siraj Gupta. I lead Harness development at Warp. Um in case you haven't heard of Warp before, um we started off by building a state-of-the-art agentic development environment. Um we have about a million active users that use that. Um it was born out of a modern-day terminal. Um and nowadays we're building a um cloud agent platform to enable teams to build software factories. Um today I'm going to talk about um self-improving software factories, which I know might sound kind of buzzy, um but I'm going to break it down and hopefully you can walk out of here with something concrete that you can implement yourselves. Um you've probably heard of both of these terms, self-improvement and software factories, used throughout this conference, but I'm sure you've heard of them independently. Um you've heard about how self-improvement is about agents being able to um improve over time and agents and models um and, you know, phasing out humans from that loop, so that way the the improvement loop can be automatic. Um and I'm sure you've heard about software factories being um the way that like the software development paradigm is shifting from individual coding agents to building out automations that take work from triage up until production. Um but less has been said about how software factories themselves can self-improve over time, so that the the factory can get better, more efficient, faster. Um and I think this is going to become an extremely important part of our jobs as software engineers as we transition from building products to building factories and and maintaining them. Um so let's dive in. Um self-improvement is a pretty broad umbrella, but I'm going to focus on three concrete ways that agents can improve over time, agent factories can improve over time. I'm going to start with um skills, then we'll talk about persistent memory, and then I'll finish off by talking about model routing. Um so skills are a great way to give your agents procedural memory. I don't think I have to kind of spell that out for you. You describe a a procedure to your agent that you expect it to have to do a task over and over. Um for example, let's say you're building a triage agent, which is a pretty common thing you might build in a software factory. Um and you give it a skill on how you want it to go about reproducing issues. Um those skills get stale over time. Um humans give feedback to the agents. Agents learn things in their own um runs and trajectories. Um and the like the skill that you previously gave it becomes dated. So, how do we fix this and let the agent improve over time using that skill? Well, one way to do it is to deploy an outer loop agent. Um and this is what we've done in our own internal software factory that's been really productive. The idea is that your um your inner loop agent is the thing that's actually applying the skill. It's like doing the triage. And then you have this other outer loop agent that is observing your inner loop agent's runs and improving its skill over time by looking for um for mistakes it made or looking at feedback that the human has given it. Um so, I'm going to show an example of that. Um let me swap here. Cool. Um so, this is actually a skill that we've used internally at Warp. We've open-sourced our client repository and it's fully run as as a factory. And so, this is an example of an of a of a skill that's that's backing a triage agent that we have today. The idea is that um you you get an issue that comes in on GitHub, and you want to automatically run an agent to figure out like what sorts of um like it what sort of information did the user not provide that is going to help us figure out if this is something that we should build, an existing issue that we might actually just want to dedupe against or something else. Um and then we have a Sorry, let me just switch over here. And then we have a workflow that actually like runs that that triage um agent. Um this is a pretty simple workflow. It just Every time an an an issue gets submitted on the repo, this um this workflow runs. And then here's the actual outer loop agent, which is about looking at the the work that the triage agent has done and improving it over time. So, the idea is that the agent looks at the decisions that the triage the inner loop triage agent made, and it um collects feedback signals, which might be uh you know, thumbs up or thumbs down on the issues. Um it might be users commenting on the issue. It might be someone from Warp um an employee of Warp commenting on the issue, uh giving the agent feedback on what it did. Um and then ultimately that leads to like some synthesis of of the of that um iteration. And so, we we end up basically updating our inner loop skill with this outer loop agents. Um and that's what leads to this self-improvement loop with skills. What's really nice about this is that um the So, in step four, you'll see we we added the triage skill, and then we open a pull request. So, that means that um all of the improvements to the inner loop skill are going to be tracked through get. So, you get like full observability into like how that skill is transformed over time. And that it also means that um a human is actually going to review those updates to the skill. So, that way, you know, this this outer loop agent doesn't make a mistake and ultimately cause your triage agent to to actually perform worse. Um yeah, so that's that's kind of what what I want to talk about for skills. Um let me talk about persistent memory now. So, I'm going to go back to my deck. Um cool. So, skills are great for remembering procedures, but what about everything else? So, like what what happens when your Sentry agent um looks for an issue and and finds an issue, does some work to like gather context about it, and fixes it, but then now you have you encounter a similar issue in the future. Your century your agent might get lucky and like be able to figure out the same root cause that it did the first time around, but that's not guaranteed. And and even if it does, you're you've probably wasted a lot of tokens re-gathering context on an issue that you've already fixed before. So, that's where something like persistent memory comes in. You can think of this as a fact store scoped to an agent. Your agent runs and similar to the skill improvement loop, you have an outer loop agent that's going to extract facts, learnings, outcomes from your from your inner loop agent. So, that way future runs of the agent can lean on what's already been done in the past. So, I'm going to show an example of this. Come back to the deck here. So, in warp or in Oz rather, like I mentioned, we built out a a cloud agent platform where you can run agents in the cloud. And for all for all these agents, you can attach memory stores to them. Again, a memory a memory store is a collection of facts. In this case, I have a century agent that has collected some memories over time about root cause analysis that it's done in the past. What's cool is like you can actually version this memory, you can update it as a human, create new memories or delete them yourselves, but primarily this this memory creation loop is driven by agents. You can also see where these memories were sourced from. So, I can actually like open up the run here to see Oh, well. I think this my IP address here keeps changing and this my our staging platform is gated to an IP address, but anyways, you should like you can see where those memories were sourced from. So, that way you can be like, oh, like maybe that wasn't actually an important memory or that was kind of like a local maxima that I don't want to include in my memory store. Um and then when your agents continue to run, so in this case I have the sentry agent that's um that's done a bunch of runs after it's created those memories, it actually can lean on those memories in the future um when it's making um when it's triaging new issues. So in this case, you'll notice that um this agent ran, it did a bunch of work to do some root cause analysis, and then it ultimately reported back on which memories it used um uh in this case it it it it was able to use those um five memories that I had in that memory sort to to speed up the work that I did this time around. Um, cool. Uh let me go back to the deck. So, what's cool in Oz is that persistent memory works across all harnesses, so Warp Zone, proprietary harness, plot code, Kodak, it doesn't really matter what you're running on Oz, um memories will automatically be created for you. Um, but you also have the control to to review those memories, version them, delete them, or edit them as you please. Um, all that being fully traceable. Um, cool. That's what I wanted to say about memory. Um, last thing I want to talk about here is model routing. So, um you may be wondering how this plays into self-improvement loops for um factories. Well, first I want to touch on why model routing is important in a factory in the first place. Um, it can become prohibitively expensive to run all of your agents that are doing simple things like triage or fixing simple CI failures with Opus. That's just like going to be really expensive over time, and I'm sure some of you have maybe been burnt by this or have heard stories of of or organizations being burnt by this. Um, so model like choosing which models you use for which tasks that is actually very important um and something that we're we've been working on at work. Um, so in Warp, you can actually use uh our out-of-the-box model routers, we call them our auto models. Um, over like as new models come out, we're all we're always evaluating like what are the best models when it comes to Pareto efficiency. And so, like if you're a new user and you just want to get started, it's much better than like pinning your model to like Opus or or or Haiku, for example. Um let me show you all um that experience in Warp. So, let me go here. Give me 1 sec. Got to drag this over here. So, here I have my Warp desktop um and you can see that um alongside being able to define models um being able to use Warp's auto models is one option, like these models here, but you can also define your own model routed um uh models, which uh basically boil down to like a set of rules that you can give um to us and we will like make decisions about model routing based on those rules. So, here I have an example of a configuration file that I've defined um where I've defined a few different rules about like I want database migrations done with with GLM fire with GLM or I want um you know, uh runbooks and API documentation done with Quen. Um So, this really allows you to like like to define classes of tasks and be able to say, "Okay, I want this this class of task done by this model because that's kind of what you've seen." At this point though, that's kind of more of an art than a science um and so uh what we're working on next is being able to define evals for um customer-facing evals where you can actually define what sorts of things you care about and what knobs you want to turn on these configurations and actually see, "Okay, is is Haiku better than fi- than GLM than Opus in a specific class of issues?" That's like specific to your workflows and not just some like generic benchmark that um that someone posted online. Um let me go back here. Give me a sec. Cool. Cool. Um and yeah, so let me uh I'm in my slideshow. So, here is kind of like a a high-level overview of how this works in Warp today. So, um internally we have our model routing rules, um but then we have like an eval sidecar that's determining um what sorts of tasks are actually better with which models. And we do this with a simple like best-at-k um approach where when you type a prompt, we run a bunch of agents in Oz uh across a different set of models. Um and we found success in like being able to determine, "Oh, like UI tasks are really well done with GLM. Um we don't really need to run those with Opus. It's much more efficient to run those with GLM." Um so, that's how we've been using it, but um sooner enough we're going to be building this into our product so that way our our customers can actually use it in their own workflows. Um Yeah, that's uh that's mostly what I had today. Um we're around in a booth around that side. So, if you're interested in this kind of stuff, come talk to me. I'm always down to talk about this kind of stuff. But yeah, thanks for your time today. That's all I had. \>\> \[music\]