Ryan Cooke, engineer at WorkOS, critiques the standard 'software factory' pattern (agent in a sandbox, prompt, merge PR) and argues output metrics like PR counts or lines of AI code measure the wrong thing—what matters is outcomes, i.e. shipping more to customers. He describes how WorkOS embeds its engineering process into the factory itself: TARS, an agent living in Slack, Linear and GitHub that uses webhooks to track projects and pick up tickets autonomously; a PM agent that drafts the 'Hilltop' product doc and breaks approved work into tickets, solving the blank-page problem; and Horizon, an orchestration layer in front of an internal MCP gateway connecting tools including Snowflake, which Cooke calls the single best investment for teams starting a factory. He also covers bug and support triage, WorkOS's own sandboxes, a company-wide memory layer, and success measurement via defect rates, recovery time and adoption. # Video Identity Observations ## Source Material Overview - **Title:** "No, That's Not a Software Factory — Ryan Cooke, WorkOS" - **Format:** Conference talk (final day of a conference), delivered by a named individual affiliated with a company. ## Explicit Self-Description - **Speaker identity:** Ryan Cooke states, *"I'm Ryan. I'm one of the engineers here at WorkOS."* - **Role within the company:** He says, *"new API to products I lead called Vaults"* — indicating product leadership responsibility alongside engineering. - **Company context:** *"We run this product engineering culture at WorkOS. This is where engineers are responsible for a lot of the product functions. We don't have product managers on teams today."* - **Openly acknowledged gap:** *"How you're handling authorization... This is something I don't talk about because we haven't figured it out yet for ourselves."* ## Intended Audience - **Practitioners already familiar with AI-driven development:** *"this is probably not a new concept to any of you. I'm guessing if you're all here, you're very familiar with software factories."* - **Teams building or evaluating software factories:** *"If you're just getting started with thinking about a software factory, it's well worth your time to invest in an internal MCP gateway server..."* - **Peer outreach:** *"If you are building software factories, I would love to talk to you."* — suggests a community of peer builders. ## Claimed Scope of the Talk - A critique of industry-standard "software factory" setups, arguing that **output metrics** (PR counts, code volume) *"can actually disguise how well these systems are working"* and should be replaced with **outcome metrics**. - A detailed walkthrough of WorkOS's internal systems: **TARS** (agent interaction layer across Slack, Linear, GitHub) and **Horizon** (infrastructure orchestration) plus an internal **MCP gateway / "context engine."** ## Demonstrated (Not Claimed) Presentation Patterns - **First-hand practitioner framing:** The talk is grounded in concrete internal examples — a Vaults API project, Slack kickoff commands, hilltop documents, Snowflake semantic tables — rather than general advice. - **Candor about limitations:** *"these agents aren't perfect. There are times where it grossly overestimates what we're trying to accomplish."* - **Thought-leadership positioning:** Open critique of industry hype (*"posts on X and blogs"* touting PR percentages) suggests the speaker/channel presents contrarian, measurement-focused perspectives. - **Networking intent:** Repeated booth invitations indicate the video is part of conference community engagement, not standalone channel content. ## Self-Check All observations trace to quoted or precisely described transcript passages; no private motives inferred and no channel-wide conclusions drawn from a single video. # Delivery Observations — Ryan Cooke, "No, That's Not a Software Factory" ## Overall Register - Conversational, first-person conference delivery with frequent verbal fillers: "um," "uh," "like," and "kind of" appear throughout. - > "we kind of very quickly kind of ran into this situation" - Hedging and self-correction occur naturally ("cloud code" vs. "Claude code," "their our software factory"), consistent with live, unscripted speech. ## Recurring Vocabulary - **"Encode"** is a signature frame, repeated across the talk: - > "take existing processes and encode them into the factory themselves" - > "we want to encode those in automation" - **Output vs. outcome** is stated as a deliberate contrast multiple times: - > "we measure instead of just code output, we're looking at outcome metrics" - Heavy reliance on **"webhooks"** as a connective technical motif: "We subscribe to webhooks through TARS," "it's listening for web hooks," "TARS through webhooks can listen." - Web-speak hedges: "kind of," "like," "right?" as audience check-ins — > "focusing on output, right?" ## Rhetorical & Teaching Habits - **Rhetorical setup questions** to transition sections: - > "So what does this actually look like?" (used twice) - **Deliberate repetition for emphasis**, with self-awareness: - > "So, say it for the third time, it's worth repeating. Sandboxes are great." - **Analogy-driven explanation**: - > "the dream of the software factory is that it gives each of our engineers a small engineering team for themselves" - > "it's that blank page problem with writing" - **Named-system teaching style**: introduces each artifact with a one-line definition — "we call TARS... a way for users to interact with the coding agent"; "this is the infrastructure orchestration layer." ## Communication Patterns - **Honest disclosure of limits**, used as a credibility device: - > "these agents aren't perfect. There are times where it grossly overestimates what we're trying to accomplish" - > "This is something I don't talk about because we haven't figured it out yet for ourselves" - **Direct prescriptive advice** near the close: - > "it's well worth your time to invest in an internal MCP gateway server" - Warm, personal closing invitation: "please come by the booth, talk to me, would love to chat." - Structured talk arc: industry context → critique (output metrics) → their alternative (TARS/Horizon) → concrete walkthrough → broader leverage (MCP gateway) → future roadmap → invitation. ## Isolated Moments (uncertain patterns) - Mid-sentence restarts and stumble repetition ("a lot more a lot more quickly," "our our plan") appear a handful of times — plausibly live-delivery artifacts, but the evidence is too thin to call a stable pattern. - The "say it for the third time" line is a single occurrence; its effectiveness as a habitual device cannot be confirmed from one talk. # Observation Record: "No, That's Not a Software Factory" — Ryan Cooke, WorkOS ## Speaker Identification - **Ryan Cooke**, engineer at WorkOS, presenting on the final day of a conference. Identified via the video title and his self-introduction: *"I'm Ryan. I'm one of the engineers here at WorkOS."* --- ## Central Thesis **Output metrics mislead; outcome metrics matter.** Cooke argues the industry's standard "software factory" pattern (sandbox + AI agent + prompt → PR → merge) succeeds at producing code output but that output metrics (PR counts, percentage of AI-generated code) can *"disguise how well these systems are working"* and make it *"very hard to distinguish whether the output is actually driving outcomes."* WorkOS instead measures whether the factory accelerates feature delivery. --- ## Positions and Reasoning ### 1. Critique of the standard software-factory setup - Notes industry excitement following Ramp's blog post (approximately December of the prior year, per his recollection) about AI-driven software factories. - Describes the common pattern: a Cloudflare-hosted sandbox, an agent (Claude Code or Open Code), a prompt, a generated PR. - **Key claim of failure:** WorkOS started with this same setup and found it *"pretty indistinguishable"* from engineers running Claude Code on their laptops — *"This was not an incremental increase or an exponential increase."* ### 2. Automate product engineering, not just coding - The fix: embed WorkOS's existing engineering processes into the factory so it performs *"product engineering work, not just code work."* - Two-system architecture: - **TARS** — user-facing agent embedded in Slack, Linear, and GitHub; subscribes to webhooks to track project progress and autonomously pick up dependent tickets. - **Horizon** — infrastructure orchestration layer (which he says resembles Ramp's Inspects and "Minions"), sitting in front of an **MCP gateway**. - **Hilltop documents** (PRDs with purpose, customer feedback, competitive analysis, design screens, milestones) are fed to a dedicated "PM" agent that drafts them and breaks work into tickets; humans stay in the loop but no longer shepherd every step. - **Qualification preserved:** *"these agents aren't perfect"* — the PM agent sometimes *"grossly overestimates"* project scope and humans must cut it back, though he frames this as still better than a blank page. ### 3. The MCP gateway as an unexpected leverage point - Called their **"context engine"**: connects to internal systems (including Snowflake with semantic tables describing product usage and customer conversations) and gives agents orientation on when and how to use each tool. - **Surprise outcome:** it became leverage beyond the factory — internal teams query it directly from Slack for data and customer analysis, and more tools are being built on top of it. - His explicit recommendation to the audience: invest in an internal MCP gateway early if starting a software factory. ### 4. Agent-agnostic tickets - Tickets and documentation are usable by other tools: **Devin** (which he notes is popular at WorkOS), local Claude Code/Opus harnesses, and human collaborators (e.g., the security team weighing in on new projects). ### 5. Future direction - Using TARS to build their **own sandbox infrastructure** (moving off sandbox-as-a-service) for deeper session control and workload portability. - Building a **memory layer** as "evergreen context" — per-person, per-team, and org-level semantics — pluggable into both the factory and other AI tools. - **Self-improvement loop:** because they own the infrastructure, they can analyze sessions to find skill gaps, obsolete skills, and areas where the agent erred. - Success metrics he cites: defect rate, time to recovery, and anecdotal evidence of engineers voluntarily moving from local harnesses to the cloud sandbox. --- ## Tensions and Qualifications (preserved as stated) - **Admitted unsolved problem:** authorization. He explicitly declines to discuss it: *"This is something I don't talk about because we haven't figured it out yet for ourselves."* - **Stated concern about risk:** *"certainly worried about... introducing instability into our products"* — mitigated, in his framing, by measuring defect rate and recovery time. - **Not a claim of universal superiority:** the initial factory output was indistinguishable from local agent use *at WorkOS* — a finding about their own measurement, not a claim about all factories. - **Iterative framing:** "say it for the third time, it's worth repeating" — he hedges enthusiasm for raw sandboxes ("Sandboxes are great. It's cool to run an agent and have it open a PR") while insisting that engineering *process* must be encoded in automation. --- ## Traceability Note All claims above derive solely from the transcript of Cooke's talk. No positions are attributed to other companies beyond what Cooke reported (Ramp's blog post; popularity of Devin) — these are his characterizations, not verified claims from those organizations. # Speaker & Provenance Record ## Video Identification - **Title:** *No, That's Not a Software Factory — Ryan Cooke, WorkOS* - **Video ID:** `01a15ca7-2853-7af4-ac4a-b19c3be93804` ## Speakers & Roles | Speaker | Role | Basis in Source | |---|---|---| | **Ryan Cooke** | Speaker / Engineer at **WorkOS** | Self-identified near the end: *"I'm Ryan. I'm one of the engineers here at WorkOS."* Title metadata also attributes him. | | Unidentified person | Possibly moderator/host (final day of conference) | Transcript opens with *"thank you for joining the final day of the conference"*, but no separate speaker turn is distinguishable; attribution uncertain. | - **Setting:** Conference talk, described as the **final day of a conference**; no event name given in the transcript. - **Role context:** The speaker presents WorkOS's internal approach ("software factory" systems **TARS** and **Horizon**) and states he leads the product **Vaults** (*"products I lead called Vaults"*). ## Referenced Sources & Third-Party Claims All items below are claims/references made by the speaker; not independently verified. - **Ramp** — blogged about their system (referred to as "inspects" in transcript) "several months ago / maybe December last year"; credited with sparking industry interest in AI software factories. - **Tools/products mentioned:** Claude Code, OpenCode, **Devin**, Cloudflare, Snowflake, Slack, Linear, GitHub, Notion, MCP (Model Context Protocol). - **Comparable systems:** Inspects and "Minions" (systems built by other companies, per speaker). - **Third-party behavioral claims:** - "A lot of other companies have followed suit" after Ramp. - "Devin is very popular. We have a lot of folks that really like using Devin." - Claims about posts on X/blogs emphasizing output metrics (percentage/number of PRs, code generated by AI). ## First-Party System Claims (WorkOS) - **TARS** — user-facing coding-agent interface embedded in Slack, Linear, GitHub; consumes webhooks. - **Horizon** — infrastructure orchestration layer, sits in front of an internal **MCP gateway** ("context engine"). - **"PM" agent** — drafts the "hilltop document" (PRD) and breaks work into tickets. - **Memory layer** — planned "evergreen context" system (future work, not shipped per transcript). - Organizational claims: product engineering culture with **no product managers on teams**; security team reviews projects via Slack channels. ## Promotional Segments — FLAGGED - **Booth promotion (end of talk):** *"We have a booth right opposite the speaking area... please come by the booth, talk to me."* Includes an explicit **request for input** on authorization handling: *"I would love to hear how you're handling authorization... we haven't figured it out yet for ourselves."* - Implicit promotion of WorkOS's internal tooling throughout the talk (TARS, Horizon, MCP gateway), consistent with a vendor conference presentation. ## Availability & Evidence Limits - **No curated metadata supplied** beyond the title; speaker identity rests on the title attribute plus self-identification in the transcript. - **No comments section status** available from the source material. - **No links, citations, or external documents** are provided in the transcript; all references (Ramp blog, X posts) are unlinked. - Demo screenshots/UI walkthroughs are referenced narratively (*"here's a project I kicked off last month"*) but no visual evidence is captured in the transcript. - Transcript contains transcription artifacts (e.g., "inspects," "Minions," "cloud code," "Tarsin," "Work OS" vs. "WorkOS") — proper spellings/names are inferred, not confirmed. - A second speaker marker (`>> [music]`) appears at the end but contains no attributable speech. Well, thank you for joining the final day of the conference here. I know it's been a long week. But I'm excited to talk about a topic here that is very near and dear to me. Software factories. Um You this is probably not a new concept to any of you. I'm guessing if you're all here, you're very familiar with software factories. You know, Ramp blogged about their inspects several months ago. I think maybe December last year. I forget exactly when. It's been a little while. And that really kind of like excited the entire industry of thinking like, oh, we can actually build software very differently now that AI is extremely capable of writing code. Um a lot of other companies have followed suit. Um and there's kind of like a standard set up here. Uh you have a sandbox that you put your code into. Um you put a an AI agent in there. Maybe a Claude code section or an open code. Um and with a prompt, it generates a PR and you merge it. Um You might be building this already \[clears throat\] yourself. We have taken a slightly different approach at WorkOS. Um and the motivation for what we are building is that we see a lot of the kind of success metrics um really focusing on output, right? Like you'll see posts on X and blogs talking about the percentage of PRs or the number of pull requests or how much code uh is being generated by AI moving into production. Um and this is just really output metrics that um you know, this can actually disguise how well these systems are working. Percentage of PRs or number of PRs may be lying in overall increase in PRs. It's very hard to distinguish whether the output is actually driving outcomes. And so that's kind of where we focus our efforts when we're thinking about a software factory. It's like if we're going to go put engineering time into building this level of automations and there's a bit of work that needs to go into this. Full-time teams dedicated to this. How do we want to think about the success of this and whether this is actually driving value for our organization, our engineering organization? And so we measure instead of just code output, we're looking at outcome metrics. And that is really about like are we accelerating our ability to deliver features? We think that the dream of the software factory is that it gives each of our engineers a small engineering team for themselves. And so the consequence of that really needs to be that we're building more and we're shipping more and that as we embark on complex features, those can get built a lot more a lot more quickly. So what does this actually look like? Well, we kind of started with the same thing you've seen in other other factories. We start with a sandbox we built on top of Cloudflare. We put an open code model router in there. We're feeding it prompts. Um, and we kind of very quickly kind of ran into this situation where this wasn't driving more outcomes. This was not an incremental increase or an exponential increase over engineers just driving cloud code on their laptops. It was actually pretty indistinguishable for us. And so we really kind of started to think about, well, how can we embed our engineering processes into the factory itself? It's not sufficient that the factory is producing code. We actually want it to take a lot of the other work that our engineers do to produce products and automate that as well. So, we kind of split this into two parts. One is we have the system we call TARS. This is a way for users to interact with the coding agent. And it is embedded into the tools that we use. So, not just Slack, but also Linear and GitHub. We subscribe to webhooks through TARS so that TARS can actually track the progress of projects in addition to generating the code outputs. And then we built a separate system called Horizon. And this is the infrastructure orchestration layer. This is a lot more similar to Inspects and Minions and and some of the other systems that other companies have built. But, it sits in front of an MCP gateway. And I'm going to talk a little bit more about why that MCP gateway has been really transformational for us. So, by feeding our webhooks, activities that are happening in other uh source code systems and project tracking systems, we're starting to get to that level of autonomy where our factory is performing product engineering work, not just code work. So, what's an example of this? In Linear, we can define tickets that have dependencies. So, one ticket blocks another ticket. Pretty common for, you know, breaking down uh large units of work into smaller ones. Because uh TARS is getting webhooks on ticket completion, it can automatically pick up the next ticket in a cycle. And so, we can do through our planning process construct like a map of how this plan we think this plan is going to get executed and TARS can start to execute on that autonomously. The other thing that we have is in between those steps when a ticket is completed, we can ask TARS "Can you reevaluate the linear project and let me know if it's missing now tickets?" Because as you do work, you're learning about where the gaps in your plan. And we want to continuously keep our our plan fresh by using the agent itself. Because the agent is determining that there may be missing pieces to what we initially planned for the project and the goal that we have in mind. So, we run this product engineering culture at WorkOS. This is where engineers are responsible for a lot of the product functions. We don't have product managers on teams today. Um and one of the rituals as part of this product engineering process is we create a hilltop document. It's a PRD and it is intended to both define what is the purpose of this project, but also incorporate um what are customers talking about? Like where are we seeing the need for this unit of work? Uh it looks at competitive analysis. Like are there similar products out in the market today that we can draw inspiration from? We start to bring design early design screens into this. Um and we outline like what are the major milestones. And so, by kind of consolidating and canonicalizing this information into a document, this is something our product engineers have been doing uh throughout the entire uh company. Um now we can ask give this resource to an agent and an agent can break this into units of work. Uh and so, this has been a really uh powerful way for us to take existing processes and encode them into the factory themselves. Like I mentioned that it's listening for web hooks. I'll show an example of what this looks like, but it it can see that the hilltop document through linear tickets has been reviewed and approved and by virtue of it being approved can start work on this project automatically. So we don't need a human to be shepherding this agent through every single step of the life cycle. So we created an agent specifically for this part of the process. We call it the PM. And it is doing the first draft of the hilltop based on the brief specification that we give it. It's adding context to that in addition to reading the human reviews and then it picks up the implementation and breaking that into tickets. And there's an opportunity for humans to stay in the loop for each part of this process. I mean it's very common for a human to intervene or comment on a ticket that's generated by AI, give it further guidance or refinements. Um but it's really about that like cold start problem where we can kind of get over the hump of creating all these different resources. Okay, so what does this actually look like? So here's a project I kicked off last month two months ago. Um we're adding a new API to products I lead called Vaults. And with a command here in um in Slack I can kick off this project with a simple description, usually a few sentences of what I'm trying to accomplish. Tarsin goes in and creates all these resources for me. I don't need to go and create the project in linear. I don't need to create a draft of the notion document. Um all of the decision logs and the open questions. Um Tars is creating a first pass at that. And so this then gives me an easy framework for me to step into as the lead product engineer and start giving more specification, rounding out areas that aren't well defined with knowledge that I had know about what we're trying to accomplish with the project, and then hand it back to Tars uh for execution. So, here's an example of uh the project. Um and it's like I said, it sets up these first milestones around our product engineering process. So, we haven't done the hilltop review yet here. When that's done, we'll uh we'll we'll mark this ticket as complete and Tars will pick that up and move on to the next stage of project implementation. So, what we found is like a lot of teams are now doing more project brief and hilltop work because our agent can seed all that information. It's that blank page problem with writing. If you can come into a document that already has information, and you know, these agents aren't perfect. There are times where it grossly overestimates what we're trying to accomplish with this project, and we have to cut out a lot of the scope that it comes up with. But, that's fine. That's a lot simpler for an engineer to add input into into rather than them spending time to set up all these primitives themselves. And then we can continue to drive this work, you know, through Slack, through Linear, um and ultimately uh results in PRs in GitHub. The other thing that this lets us do is we don't necessarily always need to use our coding agent to implement uh pieces of work. Uh Devin is very popular. We have a lot of folks that really like using Devin. They can point Devin to these tickets and the same documentation to give Devin the context it needs to go and implement different parts of the work. Uh same with the local cloud code if you're just using Opus in a local harness. Um it can grab through MCP all of these uh all of these pieces of documentation and use that as context for its work. And then, of course, we have people that collaborate in these project channels as well. You'll notice our security team will come in and look take a look at new projects, weigh in on the security implications, and so forth. So, I alluded to we we kind of early on in the the development of our factory is created our own MCP gateway. Um, we call it our context engine. Um, this connects into all of our internal systems, but it also builds system prompts and context around what how to navigate these tools and when to use these tools. So, it is connected into Snowflake, which is our main data lake. We have semantic tables that we've built in Snowflake that describe product utilization or customer conversations. And then, we can provide an agent through our MCP server in the tool description list of here are the tables, this is the content they contain. If you're trying to answer questions about this type of content, write a query for these tables. And so, it gives us a little bit of orientation to an agent to understand how to use how we use our tools, how we've organized linear in Snowflake, and give some guidance on where to find information. What has been really surprising about this is we actually now use this MCP server across a lot of other different internal tools. We open it up for people to query directly from Slack to do data or customer analysis. And so, while we initially kind of envisioned that this would just be a way to connect our agent or coding agent to all of our systems. Uh this has been an incredible piece of leverage for a lot of our internal teams, and we're building a lot of other tools on top of this MCP gateway. So, if you're just getting started with thinking about a software factory, it's well worth your time to invest in an internal MCP gateway server that both connects all your tools, has descriptions that can tell the agent how to use those tools and how you specifically organize the information in them, and I think you'll find that that ends up being useful in a lot of other use cases. Like I mentioned, we try to automate other parts of our software development. Um we have like bug bug requests that come in through Slack. Uh so, TARS through webhooks can listen to those and take a first pass at triaging and implementing opening PRs to fix bugs. Uh we have a lot of our customers uh in shared channels in Slack, as well. Uh we found TARS to be pretty useful to uh triage support requests that we're getting through them, because it can look at the code, it can often understand like where the customer might be running into problems with our product uh from the code perspective. Um I showed you an example of how we kind of organize major features around this. Uh and then the last place I think is really exciting, we're all trying to get to the self-improving software, or the self-driving software. So, we're using uh TARS to actually build out our own uh sandbox infrastructure. So, we want to move off of some of the sandboxes as a service and own that infrastructure layer ourselves. Um Some of the reasons we have for that is we want really deep control over the session information, and uh be able to move workloads around different parts of our infrastructure. Uh so what we're building out next is our memory layer and we want this to be an evergreen context uh about what every person in the company is working on, what team do they sit in, what products are they responsible for, and then at an organization level, what is the semantics about Work OS and how we do our work. And this we want to be able to plug into both our software factory, but lift that context out of our factory and into our other AI tools as well. So, I think this is kind of our our major takeaway of where we've seen a lot of folks invest effort just around outputs and where we think we can actually get that human exponential value out of our factory. Can we be actually delivering values to our customers? We think a lot about shipping and customer impact and so we want to encode that into their our software factory itself. Um certainly worried about, you know, introducing instability into our products. So, all right, can we measure uh defect rate and other um you know, time to recovery uh metrics? Um and then a lot of anecdotal, like we want to see our engineers using TARS as a sign that this is giving them value and accelerating their work. We want to see that they're moving off electing to move out of their local harness and into a cloud sandbox. And then the real goal is we can use this information because we own the infrastructure, we can see what's happening in the infrastructure. We use this information to self-improve the factory. We want it to learn, we want it to get better about how it writes code. We want to see where um our engineers can improve their skills. There's so much moving quickly in AI. It's kind of like a constant race to keep up with the latest um you know, tips and techniques. We can actually point agents at sessions and see where are the gaps and how people are using our factory. Where is there a skill that we should be building because the agent made a mistake. What skills are no longer relevant? Maybe the code has changed to a degree where a skill that we wrote six months ago is now obsolete. So, that continues verification. Semi-online is where we kind of see a big advantage to actually owning all the infrastructure ourselves. So, say it for the third time, it's worth repeating. Sandboxes are great. It's cool to run an agent and have it open a PR. We really think about our software engineering practices and processes and we want to encode those in automation. I'm Ryan. I'm one of the engineers here at WorkOS. We have a booth right opposite the speaking area. If you are building software factories, I would love to talk to you. I would really love to hear how you're handling authorization. This is something I don't talk about because we haven't figured it out yet for ourselves, but maybe you all you all have some insights. So, please come by the booth, talk to me, would love to chat. Thank you. \>\> \[music\]