Episode of AI & I featuring Scott Wu, cofounder and CEO of Cognition—the startup behind Devin, billed as the world's first autonomous coding agent, and acquirer of the AI code editor Windsurf. Wu discusses why he believes AGI may already be here, his journey as a founder, why computer science fundamentals still matter in an AI-first world, how programming and the AI-first software engineer's workflow will evolve, how Devin compares with Claude Code, and using reinforcement learning to build better coding agents. Timestamps: 00:00:00 – Start 00:02:02 – Introduction 00:02:32 – Why Scott thinks AGI is here 00:09:27 – Scott's personal journey as a founder 00:16:55 – Why the fundamentals of computer science still matter 00:22:30 – How the future of programming will evolve 00:26:50 – A new workflow for the AI-first software engineer 00:29:33 – How Devin stacks up against Claude Code 00:40:05 – Reinforcement learning to build better coding agents 00:50:05 – What excites Scott about AI beyond Cognition # Video Identity Observations: "How Cognition Built the World's First AI Coding Agent—Before Claude Code" ## Channel Presentation Evidence ### Explicit Self-Description (Creator Claims) - **Channel/show name identified as "AI and I"** — stated directly in the outro: "smash that like button and subscribe to **AI and I**." - **Host identified by name**: the outro refers to "Dan Shipper as the captain of the spaceship," and the guest closes with "Thank you so much for having me" after a host-led interview. - **Format claim: interview podcast** — the host repeatedly says "I brought Scott Wu… on the podcast" and "This is a great episode." - **Scope claim (content focus)** — the intro promises coverage of: AGI definitions, the future of programming, competitive bets by "Cognition and Anthropic and OpenAI," CLI tools "like Claude Code and Codex CLI," and the Windsurf acquisition. - **Tone claim** — the outro describes the show as "the epitome of awesomeness… a roller coaster of emotions, insights, and laughter." > Note: The outro's effusive style ("I'm absolutely hopelessly in love with you") reads as satirical/humorous patter rather than a factual channel description, but it is an explicit on-channel statement. ### Demonstrated Presentation (Observable Patterns) - **Interview format with prominent founder/CEO guests** — the episode is a long-form one-on-one conversation with Scott Wu, co-founder and CEO of Cognition, introduced as "one of the most important experts on this topic." - **Audience: technically literate software engineers and AI practitioners** — evidenced by host and guest discussing, without simplification: - CLI agents ("Claude Code and Codex CLI"), IDEs, tab completion - Post-training, RL environments, reward function curation, sample efficiency - Specific tooling like Datadog, Grafana, Kubernetes, version locks - **Audience also includes people anxious about AI's impact on careers** — the intro frames the stake: "a big question about whether we're going to need software engineers at all anymore." - **Host demonstrates personal hands-on expertise with the tools discussed** — e.g., "I used Claude Code with Opus before it came out and I was like, 'Holy—this is crazy'" and "in my testing" of Devin's persistent cloud environment. - **Host offers his own analytical frameworks** — e.g., proposing an AGI definition from child development ("economically profitable to never turn your AI off") and describing a bifurcation between "traditional engineers" and "AI-first engineers." - **Promotional framing of the guest's company** — the host cites Devin at "73 million in ARR" and the Windsurf acquisition in the intro, framing Cognition as "one of the foremost players." - **Long-form, unstructured conversation style** — the discussion moves organically across AGI philosophy, founder psychology, product form factors, and RL strategy rather than following a tightly segmented structure. ## Audience and Claimed Scope Summary | Dimension | Evidence | |---|---| | **Intended audience** | Software engineers and AI-savvy listeners (transcript's technical depth); "If you're looking for a conversation that gets down to the actual brass tacks of what the future of software engineering is like, this is the episode for you" | | **Claimed scope** | AI, AGI, and the future of software engineering (intro topic list; channel name "AI and I") | | **Format** | Hosted interview podcast with startup/AI leaders | | **Distinguishing presentation** | Host combines personal tool testing with conceptual frameworks; guest framed as "rare combination… practical, hands-on frontier experience… incredibly articulate" | ## Claims vs. Demonstration Distinctions - **Claim (unverified from this source alone)**: Devin's "73 million in ARR" and Windsurf acquisition figures are host assertions restated as context. - **Demonstrated**: The channel attracts high-profile AI-industry founders as guests and hosts substantive technical dialogue — directly observable in the transcript. - **No explicit channel-wide mission statement or about-page text appears in the source material**; identity evidence is limited to the intro/outro and demonstrated conversational style. # Delivery Observation: "How Cognition Built the World's First AI Coding Agent—Before Claude Code" ## Speakers & Format - **Two-voice interview format**: a host (Dan Shipper, per the outro) and a guest (Scott Wu, Cognition CEO), marked by `>>` speaker changes in the transcript. - The transcript also contains an **intro monologue** and an exaggerated **outro call-to-action**, likely stylized bookends rather than live delivery. --- ## Host's Delivery Patterns ### Conversational Framing & Agenda-Setting - The host opens with a **scene-setting stakes statement**: "Software engineering is being radically changed by AI." - He uses a **repeated "We talk about…" list structure** in the intro: "We talk about what AGI is… We talk about what programming is going to look like… We talk about the different bets…" - He frequently **pre-frames questions with his own observation first**, then invites the guest to react: "I've seen on my end… I'm curious if this is similar to what you're seeing." ### Validation & Building on Answers - A recurring habit of **affirming before extending**: "I think that's such a good point," "I agree with that," "Totally." - He **recycles the guest's phrases back**, e.g., echoing the sync/async framing: "there's a tight dependency between model capability and what the right affordance or what the right harness is." ### Vocabulary - Tech-flavored, current terminology: **"form factor," "primitives," "brass tacks," "full send," "AI-first engineers."** - Slightly hyped promotional register in the intro/outro: "the epitome of awesomeness," "knowledge bombs," "craving for more." --- ## Guest's (Scott Wu) Delivery Patterns ### Heavy Verbal Repetition as a Filler Habit - The single most prominent feature: **immediate word repetition**. Examples throughout: - "Yeah. Yeah." / "Yeah, Yeah, Yeah." - "but but but" — appears extremely often, e.g., "but but but but you know, from from what we can see." - "I I I think" — e.g., "I I I think we have a lot of levels of AI development." - This repetition is **stable across the entire conversation**, not an isolated moment. ### Analogies as Core Explanatory Device - He repeatedly **teaches through analogy**: - The **child-development/leash analogy** for agent autonomy (host's, which he engages with). - **Calculator protests** for AI skepticism: "teachers actually used to to to picket and protest against the idea of calculators." - **Tractor/percentage-of-labor** analogy for automation: "as soon as we had the tractor." - **"Officer's school" / investment banking hazing** for skipping junior grunt work. - **Bricklayers into architects** for the IC-to-manager shift. - **Devon as "our little buddy… a cute little otter with his own computer."** ### Hedged, Qualified Claims - Frequent softening language: "at least a little bit facetious, obviously," "I'm making fun of myself a little bit here," "maybe this is a controversial take," "if that makes sense" (used at least 4 times). - He often **answers in "long-term and short-term" splits**: "I'll give the the long-term answer and the short-term answer." ### Concrete Anecdotes to Ground Abstractions - Habit of pivoting from abstract claims to specific stories: the **Grafana dashboard eval** ("there's a graphana dashboard that you need to set up and for some reason it's not working"), the **Data Dog debugging flow**, ordering **"all of our Amazon packages with Devon."** ### Numbered/Structured Enumeration - He organizes answers into explicit lists: "the three that come to mind are one is… two is… and then three…" --- ## Interaction Patterns - **Short host interjections** during guest turns ("younger Scott," "Well, I mean, investment banking, I don't know") — casual, lightly humorous sparring. - The host **poses challenge questions** ("does that make you worried about competing with OpenAI"), and the guest responds with hedged, framework-style answers rather than direct rebuttals. --- ## Uncertainty Notes - The transcript is likely **auto-generated** (misspellings like "Cloud Code" for Claude Code, "Swinsurf/Windsorf," "touring test"), so exact word choices and filler counts are approximate. - Speaker attribution in a few overlapping passages is ambiguous, so some patterns above may mix the two voices. - Tone qualities (pace, warmth, pauses) are **not observable** from text alone; only lexical and structural habits are claimed here. # Observation Document: Cognition / Devon Conversation **Video:** "How Cognition Built the World's First AI Coding Agent—Before Claude Code" **Participants:** Dan Shipper (host), Scott Wu (guest, co-founder & CEO of Cognition) *Speaker attribution based on transcript markers (>>); some boundaries are approximate.* --- ## 1. Framing & Context (Host, Dan Shipper) - AI is "radically changing" software engineering; programming "in English" via agents is changing established techniques and primitives. - Raises the question of whether software engineers will still be needed. - Cognition builds **Devon**, an "AI software engineer" at **$73M ARR**, usable like a Slack teammate with its own computer. - Cognition **acquired Windsurf** (a Cursor competitor). - Devon's positioning note: host frames it as predating Claude Code per the title. ## 2. Is AGI Here? (Scott Wu) - Wu says his claim that "AGI is here" is **"at least a little bit facetious"** — an important qualification. - **Reasoning:** By what people 10 years ago would have called AGI — passing the Turing test, solving hard technical problems (IMO/IOI gold medals), interacting with the world — "to first order, we've basically done all those things." - Yet "there's still so much more that humans do"; that gap will persist ("Peru" [sic — apparent transcription of "true for"] quite some time). - He argues percentage-of-work definitions are ill-defined: by the standard of work done 1,000 years ago, "way less than 20%" of that work remains (tractors etc.), so automation percentages have long been passed. - **Conclusion:** There are many levels of AI development; no hard cutoff for AGI; but systems have hit milestones "people would have considered insane just a few years ago." ### Host's counter-definition (Dan Shipper) - AGI defined via **child development analogy**: the "leash" lengthens — infant → teenager → college. Agents have followed the same trajectory (tab-complete → 10–20 minute autonomous runs). - His definition: **AGI = when it's economically profitable to never turn your AI off**, and enough people do so. ### Wu's response - Calls it "super fair" but pushes back on the **economic value** criterion specifically, because economic value depends on substitutability and competitive dynamics (if everyone has an AGI, they compete with each other). - Agrees an always-on agent doing meaningful work is a real threshold. - Notes the doubling/lengthening trajectory of agent runtime "has continued" despite warnings never to trust exponentials. ## 3. Wu's Background & Motivation - Previously founded **Lunch Club**, ran it ~5 years (host had assumed Cognition was his first company). - Motivation: a chip on the shoulder plus "I just have to try" — framed as better to try and fail than wonder. - Cognition began informally: exploring AI ideas with longtime friends; committed once it became clear AI coding and RL would "take off." - Current mindset: **"leave it all on the field"** — control the inputs, not the outcome. - Wu says if everything collapsed tomorrow (AI winter, hardware issues), he'd still say "that was an amazing two years." - He codes **less** now as CEO than before; hasn't "saturated his desire to write code." - **Devon's otter logo**: informal internal symbol — "Devon is just like our little buddy... a cute little otter with his own computer." - Key claim: **"If we really felt like this was going to be the end of programming, I think we would be way less excited about this problem."** Instead, it feels like teaching a buddy to code. ## 4. The Changing Discipline of Programming ### Host's observation (Shipper) - A bifurcation: traditional engineers adding AI to existing processes vs. **AI-first engineers** who only touch code when absolutely necessary, requiring new primitives for orchestrating agents. ### Wu's response - **Calculator analogy:** teachers protested calculators; society did fine; combined human+tool capability is "much more today," though some skills (memorized multiplication tables) faded. - Future engineering education emphasizes: logical fundamentals, problem decomposition, strategic trade-offs, architecture — **less** Kubernetes debugging, obscure libraries, esoteric syntax. - He leans toward **computer science having *more* value** post-AI, not less, because humans remain at the helm making decisions grounded in CS fundamentals. - Uses the phrase **"turning bricklayers into architects."** The "insane software engineer" isn't someone who types fast — it's problem decomposition and logical thinking. - **"Hazing" observation:** many professions front-load boring work (spreadsheets in investment banking; while-loops in coding). With Devon/Claude Code/ChatGPT, you can "build something on your first prompt." Now "Devon is" the junior doing that work. - Host agrees, comparing to managers needing technical backgrounds. ## 5. Form Factors: IDEs, Agents, and CLIs ### Long-term vision (Wu) - Operating at ever-higher abstraction (assembly → Python → English). Eventually you look at your product, not your code. - The **Jarvis / Iron Man** future is "in a lot of ways correct"; keyboard and mouse may not remain the right input format. ### Near-term - Two categories: **IDE** (tab-complete, chat — making you faster with hands on keyboard) and **full agentic** (async background agents taking whole tasks). - **Prediction:** both coexist for roughly **the next ~3 years**, then everything shifts toward dictating to an agent. - **CLI agents** (Claude Code) sit in the middle — more synchronous, requiring check-ins, not fully autonomous (doesn't usually create PRs, run full tests, integrate with all systems). It's "a spectrum, not a binary." ### On tab-complete - Wu's "nerd view": **"tab is speculative execution... to coding"** (transcript renders "speculative to coding") — the most synchronous tool. ### Sync/async workflow example - Building a feature: flesh out decisions (sync, conversational, Jarvis-style), build spec/PRD, hand off implementation (async), then PR review (sync). **How to move between sync and async is "a pretty unanswered question today."** ## 6. The Claude Code Shift (Host question, Wu response) - Host: ~3–4 months ago, momentum moved "literally overnight" from Cursor/Windsurf to CLI form factors. Why? - Wu's analysis: - It's "an incredible product experience"; Anthropic "put a lot of love into it." - **Core thesis: "the correct form factor is a pretty tight function of the capabilities."** Full autonomy in the GPT-3.5 era would have failed; Opus-generation models made it viable. - **Controversial take (self-labeled):** "I actually don't know that CLI itself is the most important part." - Internal Cognition debate (IDE vs. Slack vs. web app) resolved as: **"the form factor just is a software engineer"** — the question is what the tools do for you, not where they live; tools will be callable from anywhere. - Paradigm contrasts: Windsurf tab completion **augments you**; Claude Code **hands the reins of your computer to your AI buddy**; Devon is a software engineer next to you with its own VM and repo. ### Host's addition - Claude Code was a "full send" to agentic-first engineering for one's own machine; running bash locally makes it more extensible/customizable than ephemeral environments. Notes Devon's environment is more consistent but harder to customize than your own machine. ### Wu's "hyperspace" framing - Design dimensions: sync/async, local/remote, IDE/elsewhere, single/multiplayer. Devon, Windsurf, and Claude Code each occupy different points; many points will coexist; a full suite should span them. ## 7. Devon's Bet & the Windsurf Acquisition - Devon: a persistent cloud-based agent on its own computer, reachable anytime — a different bet than other big companies. - Trade-off per Wu: like **onboarding a software engineer**, but then it can run tests itself and do tasks "basically nothing else out there can do." - Pre-acquisition, Cognition was already exploring more accessible/synchronous experiences; Windsurf was "a great opportunity." - Strategy: **Windsurf as fast time-to-value** entry point; users learn the cascade agent and DeepWiki indexing, then naturally graduate to the more async Devon flow. - He adds: human engineers themselves must learn to work with async agents. ## 8. Models, Post-Training, and RL Environments - Host asks whether lacking own models worries Wu given capability–harness coupling. **Wu does not directly answer the "worried about competing with OpenAI/Anthropic" question** — he pivots to post-training. - Cognition **post-trains (fine-tuning, RL)** but does **not pre-train base models**. - Post-training examples: teaching Devon to predict its own confidence; the **Datadog → logs → debug → edit** flow that "models don't just learn on their own"; even legacy **COBOL** scenarios. - Startup edge: **speed and focus**; Cognition's DNA is "understanding the nuances of real world software engineering and teaching that to the models." ### RL philosophy (Wu) - "The platonic ideal of RL is that you can go and solve any benchmark... then the question is just like, okay, what's the benchmark?" - Application-layer companies' job: define exact tasks, environments, tools, success criteria — then train a model that "just does it." - **Grafana eval example:** fix a broken dashboard setup (version-lock issues, cascading errors). The eval asks what the dashboard says — impossible to answer without actually getting it running → **tight reward signal**. In real-world continual learning, a wrong number on a running dashboard is much harder to detect as failure. ### Three data sources (Wu's taxonomy) 1. **Pre-training** — data exists in the world; Wu thinks capabilities here are roughly converging. 2. **Curated RL environments** — "actively the world that we're all in." 3. **Continual learning** — the long-term path; Wu believes "we will get to" it. - Key synthesis: **2 and 3 share a primitive** — building real agents, environments, and tooling. Route 2's advantage is a tighter curated reward function. - Host notes continual learning arguably already happens but is very sample-inefficient. ### Generalization concern (Host) & Wu's answer - Host worry: are specific evals brittle (e.g., Grafana changes its system)? - Wu: (a) yes, you need many environments; (b) **it depends on task setup** — recalling packages from memory isn't generalizable, but a task requiring Googling, reading docs, reading logs, and reading files **is** generalizable: "we as humans all figure it out somehow" by interacting with the real world rather than pulling from memory. ## 9. Personal Agents (Wu, outside Cognition) - Surprised mass-consumer personal agents haven't happened yet: phone-based agent that books reservations, handles messages, confirms deliveries, orders Amazon packages. - Capabilities "feel like they're there" — "maybe I'm wrong" (qualification). - Anecdote: **Cognition now orders all their Amazon packages using Devon** (memory + stored keys + browser make it possible), though Slack/Linear is "not the right form factor" — somebody else should build it. - Prediction: within 12 months this category will exist. ## 10. Unresolved Tensions & Qualifications Preserved - Wu's "AGI is here" claim is explicitly **facetious in part**, yet defended on definitional grounds. - Wu **pushes back** on the host's economic-value criterion while endorsing the always-on-agent threshold. - Wu **declines to directly address** whether lack of pre-training creates competitive risk versus OpenAI/Anthropic. - Form-factor boundaries (IDE vs. agent vs. CLI) are stated as a spectrum, not settled categories. - The sync↔async handoff problem is called "pretty unanswered." - Personal-agent excitement carries an explicit "maybe I'm wrong" hedge. # Speaker Attribution & Provenance Record **Video:** How Cognition Built the World's First AI Coding Agent—Before Claude Code (ID: `01a0f15a-d9b1-7239-9c98-8bed7eb9e175`) ## Speakers & Roles - **Host / Creator — Dan Shipper** - Named explicitly in the closing segment: *"a journey into the future with Dan Shipper as the captain of the spaceship."* - Conducts the interview, provides the framing introduction, asks questions, and offers his own views (e.g., his "child psychology" definition of AGI, his observations on AI-first engineers). - **Guest — Scott Wu** (transcript renders as "Scott Woo") - Identified by the host as **co-founder and CEO of Cognition Labs**, makers of **Devin** ("Devon" in transcript) and acquirers of **Windsurf**. - Previously founded **Lunch Club**, which he ran for ~5 years (per his own statement). - **Unnamed promotional voice (end segment)** - Delivers the like/subscribe pitch for the show **"AI and I"**; addresses Dan Shipper directly. ## 🚩 Promotional Segments - **Closing outro** — explicit like-and-subscribe promotion for "AI and I," including hyperbolic praise and a personal declaration toward Dan Shipper. Flagged as promotional, not substantive interview content. - **Intro monologue** — host previews episode topics and praises the guest; partly promotional framing for the podcast. ## Referenced Sources & Third-Party Claims - **John Collison's podcast** — Scott's claim that AGI is "here" originated there (host's reference, external source). - **Cited third-party entities/products (claims about them are speaker assertions, not verified):** - **Anthropic / Claude Code** — described as a successful CLI agent form factor; Opus 4 credited with enabling it. - **OpenAI** — Codex CLI, GPT-3/3.5/5, IMO/IOI gold-medal work. - **Cursor, Windsurf** — editor-based AI coding tools; Cognition acquired Windsurf. - **Datadog, Grafana, Cobol, Linear, Slack, Amazon** — referenced within workflow/eval examples. - **Business figures** — "$73 million ARR" for Devon (host's claim in intro); Windsurf acquisition (both speakers). - **Historical anecdote** — Scott's claim that teachers protested calculators (unverified third-party claim). ## Provenance Notes & Uncertainties - Transcript contains phonetic/name errors: "Scott Woo" (likely Scott Wu), "Devon" (Devin), "Cloud Code/Codeex" (Claude Code/Codex), "Windinsurf/Swinsurf" (Windsurf), "John Collisonson" (John Collison). - No curated metadata was supplied; speaker identities rest solely on the transcript. - **Comment availability:** not stated — no information on whether viewer comments are open. - The transcript alternates speakers without labels; attribution above is inferred from conversational flow and the explicit interview structure. Software engineering is being radically changed by AI. Being able to program in English by launching agents to build features and fix bugs is changing all of the techniques and the primitives and the best practices that if you're a software engineer, you've grown up with. And for a lot of people, there's a big question about whether we're going to need software engineers at all anymore. That's why I brought Scott Woo, the co-founder and CEO of Cognition Labs on the podcast. Scott is one of the most important experts on this topic because cognition builds Devon, an AI software engineer. Devon came out just a couple years ago and it's already at 73 million in ARR. You can think of it like another teammate that you can talk to on Slack that has access to its own computer and can build small features or fix bugs basically autonomously. Cognition also just acquired Windinsurf, a cursor competitor. So in addition to having a fully autonomous AI software engineer, now they have a programming editor with AI built in. So they're one of the foremost players shaping the entire landscape of what programming is going to be like over the next 10 or 20 years. And that's what I talk about with Scott. We talk about what AGI is and whether it's here or not. And we talk about what programming is going to look like now and into the future. We talk about the different bets that players like Cognition and Anthropic and OpenAI are making to win the programming and AI race. We talk about the the recent wave of CLI tools that have taken off like Cloud Code and Codeex CLI. And we talk about how Scott sees Devon fitting into the landscape and why he acquired Windsurf. This is a great episode because Scott is that rare combination of guests that both has really practical, hands-on frontier experience because he's building and he's also incredibly smart and incredibly articulate. If you're looking for a conversation that gets down to the actual brass tax of what the future of software engineering is like, this is the episode for you. Enjoy. \>\> Scott, welcome to the show. \>\> Yeah. Yeah. Thanks for for having me. \>\> Of course. Uh, so for people who don't know, you are the co-founder and CEO of Cognition. You are the makers of Devon, the AI software engineering agent. Um, and recently the acquirers, the lucky winners of Windsurf. Uh, uh, it seems like you've had a really crazy couple months over there. \>\> Yeah, it's been a fun few months for us. I was just going to say, I mean, it's it's been an interesting time for everyone, I think, in the AI coding space, but, uh, but but a crazy few months especially for us. Um, so the thing I want to start with first is you said something on John Collisonson's podcast recently that you think AGI is here. Explain. \>\> Sure. No, I mean so so it's it's at least a little bit facicious obviously and and but but I think it's worth thinking about that you know 10 years ago what what would we have called AGI, right? And we would have said okay it's got to be able to pass the touring test. Obviously, it's got to be able to converse with you and just like, you know, come off as a human and and and and just be able to relate to you and think and, you know, in much the same way that a human does. It's got to be able to solve like, you know, tough um technical problems. It's got to be able to do a lot of the same tasks. It's got to be able to interact with the real world the same way that a human would, right? Um and I think to first order, we've basically done all those things. I mean passing the charting test we've obviously done um you know we have uh you know open AAI and others you know have released work on you know getting a gold medal at the IMO and the IOI you know solving incredibly hard technical problems um building agents that can go and actually interact and reason in the world and obviously um you know I think there's an interesting question of well there's still so much more that humans do and then there's so much more to it um and I think that's true maybe maybe one way to put it is is you know my view is that's going to be Peru for for quite some time to come. Um and sometimes people ask about AGI from a perspective of like u well AGI is when humans have nothing left to do at all you know or something like that or you know one of the definitions that that people use is like AGI does 80% of of of knowledge work or or things like that you know I I I think these things are really hard to define and really hard to kind of exactly clarify because humans specifically do the parts that are not automated right I I mean it's it's kind of like like it it whether 80% % of human work has been automated. You know, I claim that it already has been a long time ago actually because as soon as we had the tractor and as soon as we had, you know, it's it's like if you think about what people did a thousand years ago, we are doing way less than 20% of that work, right? Um a and so so a bit facitious for sure, but I guess my point is like, you know, I I think we have a lot of levels of AI development that occur. I'm not sure there's one hard cut off on what counts as AGI, but I think it's also very clear that, you know, we've hit a lot of the things that people would have considered insane, you know, just a few years ago. So, \>\> I think that's such a good point. Like, that's why I hate the 80% of knowledge work definitions because knowledge work changes. It's not like it's not a static thing. Um, and I think people underestimate the once you once you automate one level of work, there's always another level of work above that. Uh, and I mean, we've all seen this over the last three years. Uh a lot of the stuff that I do today is uh a lot of the stuff I was doing three years ago is automated now and I'm just doing more like more per unit work which is really interesting. Yeah. \>\> Yeah. And as humans we're always so you know we're we're humanentric in our view I guess is a way to say it. We're very proud of our ourselves and our work, which we should be, but but you know, you can imagine at some point it's like you'll just be able to just like think a couple thoughts and then have all of this come out and and happen in reality. And we'll still be saying, "Oh, you know, well, AI can't do that." You know, the humans are still doing all the important work, right? And of course, it's like AI, you know, at that point or or technology in general at that point will have made us like 10,000 times, you know, faster by virtue of doing 99.99% of the work. But but it's kind of it's it's very hard to define what what counts as percentage of labor, right? \>\> Yeah. Um so I have a particular definition of AGI that I'd love I'd love to battle around with you. Feel free to criticize, poke holes in it, but also I'm interested in what you think. So um the the definition of AGI that I like actually comes from child psychology. So when children are born, they are essentially ex like totally dependent on their caregiver. you can't leave them alone for any length of time. Um, and as they get older, you can leave them for progressively more amounts of time to be on their own. So, you know, an infant or like a toddler, you can leave for like 5 minutes or 10 minutes or something like that in their room. Um, as they become children, you know, you can leave them alone for like hours or more. As teenagers, they go away for, you know, maybe a night at a time and then they go to college and they're like fully fully autonomous. Um and so my definition of and and I think if you look at the development of uh agents uh and just AI in general, it has followed that same trajectory. So when GPD3 was first on the scene, we were just at the tab complete, uh, you know, level of autonomy and now we're seeing, you know, uh, Devon or GPD5 or cloud code run for, you know, 10 or 15 minutes, 10 or 15, 20 minutes at a time. And you can sort of see this smooth lengthening of that leash in the same way that you see a smooth len lengthening of the leash um, for children. And so I think a good definition of AGI is when it is economically profitable to never turn your AI off. It's always working. It's always doing something. Um and when enough people are doing that, I think that counts as AGI. \>\> Yeah. Yeah. I I I think that's super super fair. I would say uh one one thing is obviously it's very dependent on dynamics in the sense that well if everybody has an AGI, you know, then the AGIs are competing with one another for their, you know, usefulness or something, right? Right. And so there's like some amount but but but I I I think that's right. Yeah. Like I think there is a point when you can truly just have an always on agent that's going and doing meaningful work and um and producing value off of it. um economic value is I think the the the one thing I would push back against is the idea of economic value just because as we're saying so much of economic values depending on you know how substitutable it is that you're providing you know um or or or things like that. Um but uh but no that's cool. Yeah I I I feel like the to your point about like the the the doubling time of of how long these agents can operate. I mean it's like it's it it has continued. It's it's it is insane how long that trajectory has continued. You know, it's like there's always this saying of like, you know, you can never trust an exponential curve or you can never keep predicting points on an exponential curve and yet like they they they've kept coming. So, \>\> they they certainly have. I I'm curious actually, one of the things that that this makes me think about is um because I'm thinking about growing up and I'm thinking about the process of uh growing up for you. Uh I think that Devon is your first cognition is the first company you started and is it \>\> so I actually I I started a company before this. It was called Lunch Club. Um so I ran it for about 5 years. \>\> Okay. So you're okay. So you're more than more of a veteran than I than I thought. Um \>\> well it's you know still still very much a noob. So \>\> I am curious though like what is what has that been like? You've been running Devon you've been running Cognition for at least a few years. um what were you like when it started and what did you believe about yourself in the world and what are you like now? \>\> Yeah, you know it's it's honestly crazy. Um I I I think especially be because I had run a company before and in the sense of like you know I I think over the last decade or so you know there are a lot of different you know great companies that got built but but you know the the pace of what's happened in AI and and the trajectory of AI has has already I think been very vastly different from a lot of that and it's already gone much faster. Um, I think for me, um, there were there were a few there were a few elements for sure. There's a little bit of the chip on the shoulder of, you know, I I I feel like I could do better. I feel like I could do more. But honestly, I think there was also just a feeling of like I just I have to try is kind of how I thought about it at the time. Like I I I I thought about it. The way I thought about it was if you try and you know you build something really meaningful and it works out great obviously that's great you know but but but you know what happens if that doesn't happen right and and the question for me was kind of like would you really would you rather try um and give it everything you had and just find out that you know it didn't work out and you weren't the one and whatever else or would you rather not try and wonder about whether you could have done it and I I think for me the answer was pretty clear that I just wanted to to to give it a go. Um and so that's a lot of what it was like for me. Um uh in practice obviously it it was like it was almost kind of like walking our way into building a company. You know it's it's it's not even necessarily the case that we were saying like oh we're going to build a company you know I think at the time it was really just like exploring ideas in AI and looking into the things that we found really interesting which naturally we're just nerds and so AI coding is like the coolest thing right and so we were messing around with a bunch of these you know it was me and a bunch of my friends who I've known for like years and years. Um, but as it became more clear that hey AI coding is going to take off and stuff like RL is going to really work you know over the next while and it's going to unlock a lot of these product experiences like I think there's a real question of like is this the thing that we want to commit and spend our you know all of our focus and all of our effort on um and and that was the trade-off for me. \>\> Yeah. And how are you different now? \>\> Yeah. And now I think um I've been having a great time to be honest. It's it's and I I I consider myself very lucky, you know, in that. And I I think there's like um there are a few things I think um about it that are really nice. I think obviously the problem that we work on is great. I think people I think of the people that you work with as the most important thing. you know, someone gave me this advice a long time ago that you're going to spend most of your time working and so you might as well work with people that you really like. Um, and and I still I I still always thinking about that. Um, and and so like the um no, a lot of the different, you know, I think I've learned more things. I I hopefully I've gotten a little bit better over the course of the last couple years. Um, but but but in a lot of ways, I think it has been much the same way. You know, I I I think that Do you know the line of like um you know, leave it all on the field. Yeah. And I really like that mindset of like you want to go and try your best, but you also want to be able to live with the outcome. And I think that's how we very much think about it today. Uh of like we we give it everything that we have. We we we do the best that we can. You know, you can control the inputs. You can't control what happens at the end. And and we we just want to be able to say that we gave it everything we have. I feel that. I mean, I definitely feel like inside of every I just love the people that I work with and that makes it so much better to run a company when you're just everyone's having a great time together, you know? Um, and I think another way to frame Leave it on the field that I've I've felt myself is I would do this even if it failed. Um, and even if it didn't make a lot of money and I think that's actually somewhat rare today. Um and and and it sounds like that is a big part of your journey too because these are things that you're just interested in and playing around with yourself anyway regardless of whether it was going to be a gi gigantic company. \>\> Yeah. It's it's like you know everything could blow up today. you know, who who knows what happens, you know, all all the that we have a huge AI winter or, you know, something happens with the hardware or whatever and and then, you know, AI collapses and I I would still, you know, and our company collapses and whatever, you know, god forbid and I would still be like that that was an amazing two years. Like I had a great time. I I'm really happy that I got to do this for sure. H but how does it how has it changed your relationship to something that you used to love and just play around with having the amount of responsibility that you currently do to do that thing? \>\> Yeah. Um yeah, that's a good question. I mean it's I grew up doing a lot of coding obviously. I mean funnily enough, you know, despite all of these arguably, you know, because of because of my day-to-day work, I I get to do less coding now than I did before. And so so I I don't know exactly what that implies, you know, and I obviously like like to do more coding whenever I can. Um but but I certainly have not gotten to the point of saturating my desire to write code, if that makes sense. I do think there's something really satisfying about um just building the like, you know, it's it's funnly I I think one of one of the things that comes to mind here for me is like um is the the Otter logo of Devon. Um I don't know if you've seen the Otter like on Twitter or on things like that. It it is it has always been an informal logo of ours. There's a question of whether to make it the official logo, but but like the I know random company stuff, right? But but but like the reason I say it is because um it actually is really how we think about it internally, which is Devon is just like our little buddy, you know, like a little, you know, cute little otter with his own computer and he's just like typing away and and and doing tasks for you. A and that's how kind of how we've always thought about it. Like I I think if we were I mean we we are all programmers ourselves obviously you know if we really felt like like this was going to be the end of programming I think we would be late way way way less excited about this problem. Um and I think instead it's it's it's it is kind of just like teaching your own buddy how to code and and then starting on that journey uh in a way that's been really fun for us. I think this is that's a really good segue into the thing one of the things I wanted to talk about is just how you see the discipline of programming changing. I think I've seen on my end and I'm curious if this is similar to what you're seeing. I've seen on my end there's this um obviously there's people who don't want to use AI at all but then um there's this sort of bifurcation between I think more traditional engineers who are who are adding AI into their existing processes and then there's like kind of AI first engineers who are who maybe only learn to code with AI or maybe they are senior engineers from the past but are just like going fullon into into AI and they're they're AI first um and they're only touching the code if they absolely absolutely have to which is a very different mindset. Um and I think that group of engineers has there it's there's a whole different set of ways of thinking and different primitives for how to do good engineering if you're only orchestrating agents. I'm curious if that's what you're seeing or how you see the landscape evolving. \>\> No, I think that's definitely the case. Um, and it reminds me of like like um, you know, one of one of my favorite fun facts actually that I like to share is do you know that teachers actually used to to to pick it and protest against the idea of calculators? \>\> I did not know that. \>\> So when calculators first came out, there were a lot of protests of like, you know, we can't have this. This threatens math education and all that good stuff. And, you know, obviously I mean, we did okay as a society despite having calculators in our lives. And and and I I guess my the point that I want to make with it is like sure I think there are some things that go away and and and you know maybe people are for example maybe people have their multiplication tables memorized a little bit less you know in the postcal era but obviously you know if you just look at the combination of humans with the tools and what they can do you know the answer is much much more today. Um and so I I I think what is going to happen or you know what's kind of actively happening already is I think there's going to be a somewhat different education path for um you know how how to be a really great engineer in the post AI age. Um the thing that's interesting is like there's so many levels of AI improvement that are like actively happening right now that kind of change that answer. Um but but but but you know from from what we can see I think we can kind of imagine that a lot of what that looks like is more about really deeply understanding you know logical fundamentals being able to break down problems and articulate the answers to them. you know, being able to think about different strategic trade-offs, thinking about architectures and so on, right? Um, and less about, you know, just going and debugging your Kubernetes or, you know, knowing all of these kind of obscure libraries or understanding, you know, some very particular like esoteric syntax or something like that, right? Um, and I think um and I think that trend is is already happening, you know, and so so I I think the like um some some people say that that means that like computer science has no value. Some people say that computer science has way more value. I I I I tend to think it's more of the latter. Um and the reason for that is because obviously you are still the one at the helm making the decisions, right? And a lot of how you make decisions and how you decide what to build and how you think about, you know, the trade-offs that you're making is, you know, it all goes back to computer science fundamentals. \>\> Yeah, I agree with that. I think um if you want to make the analogy and I think the it is a good analogy to make to say well you're becoming a manager instead of an IC when you uh when you use these agents um the best managers are typically have if you're pro if you're engineering manager techn typically have technical backgrounds or the best CEOs too for software products for example tend to be able to like go deep into how everything works to help resolve issues um and and also to have good expectations set for what what can be done \>\> yeah it's like turning brick ers into architects is one of the things that we've said as well. And yeah, it's like to your point, it's you know, if anything, the technical architect at the company is actually like the sickest software engineer, you know, it's not somebody who's just like walked in and right like it's like, you know, if you say like, oh, this person like insane software engineer, like what do you mean? Usually you don't you don't mean that they like type really fast, you know? What you mean is like they they can break down problems. They just like have a really great feel for things. They just like think really logically. They cover all the cases, right? Um, and I think those are the same skills that you're going to want to have. I think the thing that's kind of interesting is, and this is, by the way, not true, not just in code, but but but you know, I would kind of describe it as a lot of professions today, you know, there's for for people who just get started, there's almost like a hazing experience where you like spend your first few years doing the most boring stuff, right? And then you get to graduate and do the interesting stuff, right? And and I think what we are going to have now is a little bit more of like an officer's school of like you know going straight into learning the interesting things. And if anything that's probably more true in like you know what I mean like it's like it's probably more true in like investment banking or something like that that like you know you spend like your first three years you know just going and and and going going through spreadsheets right and then you can see some of the cool stuff. But but in code obviously yeah \>\> I think it's like it's like uh it's it's not even in in coding like it's maybe less intentional hazing but you have to go through \>\> six months of um you know learning what a while loop is and what an if statement is before you can build anything interesting and with Devon or cloud code or chbt you can build something like on your first prompt and that's a huge huge difference. \>\> Yeah. Yeah. Yeah. I think it's like it's it's like and I think it's not intentional hazing, you know, anywhere or at least in most places we like to think. \>\> Well, I mean, investment banking, I don't know. \>\> Perhaps I was going to say it's kind of like like it's it's like there's a lot of this work that has to be done and like somebody has to do it. And so it's kind of like, you know, naturally it ends up being like the most junior team members that go and have to take it on. But but but now that's Devon, right? And then you kind of get to, you know, it's it's it's to your point, it's like kind of like skipping one of the runs of the ladder and being able to be a manager directly and being able to be like an architect directly. Yeah. Yeah. \>\> Totally. Well, but I want to I want to get deeper into this. So, you know, we're talking about what what does the future of software engineering look like? What does the future of the of the landscape look like? And in particular, what I'm really interested in is the dayto-day of what software engineers are doing in this new world. And I I'm curious um for specific because I think the best way to to think about this stuff is just look at what people are doing right now because there are people who are living that way right now. I assume some of them are inside of cognition. I assume some of them are your customers. So what I want to understand is is what does that actually look like and what are the new interesting things you're learning about the way that engineering works from this perspective? \>\> Yeah. Yeah, for sure. Um yeah, so so I'll give the the long-term answer and the short-term answer. I I it's this my favorite topic to talk about by the way is like what is the future of software engineering because you know it's it it is I think still a pretty open question. Um I I think in the long term I think it's very clear that obviously these systems will continue to get more powerful and and a lot of what that looks like is just you as an engineer being able to operate at higher and higher levels of abstraction right and you know it's it's kind of in the same way that we made the jump from like assembly to like Python or JavaScript or something. it's, you know, we're going to make that leap from like looking at a bunch of boilerplate Python code to just being able to express your ideas in English of what you want to build, right? Um, and so at some point you're not looking at your code. You're just like looking at your own product and you're, you know, I I I actually think the Jarvis kind of Iron Man style future is in a lot of ways correct in terms of what we'll have like like a lot of the interfaces are going to change pretty significantly when you have like an intelligent agent that can go and execute tons of things for you and you can just go and work with all these things. like it's not obvious that keyboard and mouse is necessarily the right uh input format, right, in that world. Um and so so that's like the long-term future. What does that mean for us today? Obviously, you know, we're not all, you know, working with our own personal Jarvis quite yet today, right? But but but I think in code, um you kind of see these different form factors emerging, right? And I think, you know, the the older one that has existed is, you know, what I'd call kind of like the the IDE category, right? of basically making you faster when your hands are on the keyboard, right? And and that's all the tab complete and the chat with codebase and and all the tools there. And then, you know, the newer school is kind of this fuller agentic thing, right? Running agents uh asynchronously in the background having them take on full tasks. And um you know a a and the simple way to describe it is up until the point where the agents are capable enough to to handle everything and let you just operate 100% in that higher layer of abstraction. You want to have both, right? Because you want to have agents for the things that they can go and take on and just do it entirely independently. And then you want to have the synchronous IDE experience for the things that really need you at the wheel. Um and I would guess that that phase lasts for about like let's call it like three years or so you know like for the next three years we will have both ideides and agents and then you know at some point beyond that it's kind of like everything will just be dictating what you want to to some kind of agent form factor obviously it's not the again there there's a cut off of like well what counts as an IDE you know and what counts as an agent and you know the the interfaces \>\> yeah when you talk about for example cloud code or sorry when you talk about for example like the IDE or background agents. Are you counting cloud code as a background agent? Like where does where does the new CLI all the new CLI stuff fit into this worldview? \>\> Yeah. So it's all a spectrum for sure. And I think about these CLI agents as it certainly somewhere in the middle. I I would describe it as a bit more it's a bit closer to like a synchronous agent, right? And so, you know, if you compare it to like a cascade or something in Windsurf, like it operates a little bit more like that where um yes, it is an agent that can do multiple steps, but one um you know, you're kind of meant to be checking in with it more deeply, right? And two, it is not fully fully autonomous in the sense that it doesn't go all the way and like, you know, create pull requests for you and work with all of your systems and like, you know, usually doesn't go and test all your code for you or things like that, right? Um and and so so so it's it's more a spectrum than a binary for sure. Um I I think in general you know I think we will be kind of operating on the spectrum and things will be gradually shifting more and more agentic and more and more autonomous but we will have the spectrum at least for the next couple years. Um and uh and and then I think the natural question is like what that experience should look like of using this suite of tools across the spectrum right and so you have like the full like synchronous things like I think tab complete itself is probably the most synchronous thing. It's it's like you are still really, you know, going and dictating every line of code. You know, tab is just like helping you go a little bit fat. Tab is kind of speculative to coding is is my is my nerd view on it, if you know what I mean by that. And then it goes all the way to these like full autonomous agents. Um, and I think having the suite of tools is great, right? And then the question is kind of like how do you split up your tasks into which ones should go into which buckets and also for for more complex tasks that are going across buckets like how do you use these tools in tandem with one another right um and I honestly I think that's a pretty unanswered question today frankly like I I I think there are um you know and I think there's a lot of progress that different folks are making the space I think there's a lot of really um you know really great thinkers in the space but but I think how do you start with a synchronous experience and then hand off to an async and go back you know a simple example I might give is like um a lot of what you want to do let's say like in in the world today you know with humans you sit down and you're like all right I have this project idea I think we should build it right like what's the first thing that you do you know it's it's not necessarily I mean hopefully it's it's not that you just immediately sit down and start typing code right like a lot of it is like you're just fleshing out the details like thinking about um you know all the decisions that you're going to have to make all right like we should build this new feature And this feature is only going to imply to users in X, Y, and Z buckets. And you know, if they're in bucket X, then they already have something that looks like this. And so, you know, we need to go replace that in the UI with with the new thing that we're trying to build. If they're in Y, then it's like a totally fresh thing. So, we should walk them through the onboarding. You know, you're you're you're fleshing out all the details uh of what you need to do, right? And then, you know, I think the next step is is typically something like building out a clear spec, right? Like thinking about the technical details and building out um all of that, right? And then you know depending on the cycle obviously there's there's there's more things and then at some point it's like kind of handing off the implementation and I guess my point is I think AI coding agents should be able to help you through the full cycle or like this kind of suite of of of things should be able to help you through this this whole thing right um but but naturally there are parts of it that you want to be doing synchronously and parts of it that you want to be doing asynchronously right so going and making these decisions of how you handle users in different buckets or whatever is something where it's like you should be able to have the Jarvis experience experience of talking directly, you know, live with your agent and your agent is like, "Oh, by the way, yeah, there's this if case, there's this case. There's this case. How do you want to handle each of these? Let's talk it through together. By the way, like, you know, I looked deeper into the code paths and here are the things that I found, but like obviously the decision is yours ultimately, right?" And then, you know, maybe there's other things of kind of just like building out the PRD itself, right? And then at some point when you're going and doing the literal implementation of the code and testing and just like making sure that works, like that's something that probably should happen more async, right? like you don't have to be involved once you've kind of fleshed out all the details uh in the actual implementation itself, right? Um and then maybe when you get some comes comes time to do the PR review cycle, then you want to be synchronously there. You want to be able to read the diffs, right? A and so there are a lot of these kind of like uh these flows in our work today where naturally you want to be able to go from like sync to async to sync to async, right? Um, and I think there's there's still a pretty open question of of how you should be kind of working between those together. One thing I'm curious for your take on is I I think the AI coding space had this really interesting shift like 3 months ago where prior to three or four months ago, everybody at every including me, we were all using cursor and windsurf. Um, and then Cloud Code came out and I mean I used I used Cloud Code with Opus for like before it came out and I was like, "Holy this is crazy." Um, and literally overnight everybody at every and I think a lot of people around the space switched to these new CLI form factors. What do you make of that? Like why was why did that happen? Um, obviously still people are still using Windsurf, they're still using cursor, whatever, but the a lot of the momentum went to CLIs. Why do you think it was so successful and and what do you make of it? \>\> Yeah. Yeah. Well, first of all, I think it's it's an incredible product experience to be clear. Um, and and I think Anthropic has done very very well with that. I think um I I think there are a few things going on here, you know, but but broadly the way I would describe it is the capabilities, you know, well, the the capabilities change like every week, honestly, but there's I would say relatively meaningful step function changes every few months, let's say. And the thing that's really interesting is the correct form factor or like the correct interface is a pretty tight function of the capabilities, if that makes sense, right? like you you if if you were trying to do full autonomous things, you know, I I'm making fun of myself a little bit here. If you were trying to do full autonomous things in the GPT3.5 era, you know, it's like there there were things that you could do, but obviously \>\> younger Scott. \>\> Yes. Um um and it's obvious, you know, there there are a lot of kind of like different versions of this that that happen, but but I so I think there were a few few shifts that happened, right? I I think the capabilities like really improves to a point where you could start to handle things um you know a lot of day-to-day things in a more autonomous way for one and then I think the other thing is is just like I I think anthropic very clearly put a lot of love into the experience you know I actually think of it as as I think a bit less so something I'll say um maybe this is a controversial take I actually don't know that CLI itself is the most important part of the experience um and I I would claim you know the the reason I say you know we we have this debate actually internally at cognition of like like what is the form factor you know like should the form factor should it live in the IDE should it live in slack should it live in you know should it be its own web app or whatever and I think the answer that we kind of come to is like you know the form factor just is a software engineer if you know what I mean by that and it's kind of a non-answer but also kind you know hope hopefully it does say something which is which is basically like it's less a question of well where do you go to interact with your tool tools. It's more like a question of like what do the tools do for you, right? Um and and how do you expect to work with them? Um and so I think from that perspective, you know, engineers are spending a lot of time in the terminal. They're also spending a lot of time in the IDE. They're also spending a lot of time in Slack. You know, all all of these are reasonable things to think about and and to work with. Um and I think as time goes on, I think probably a lot of these tools will be integrated with more and more of these and so that you can kind of call them from anywhere. Um, but I think the bigger question, which I think um, Claude Code took like a bit of a different spin on is how should you be working with the tool? Um, if that makes sense. And and and I think um the you know the the the way that I would describe it is kind of like their view of the tool is like the tool is you you know whereas whereas like you know if you're in windsurf um and you're doing a tap completion it's like um this is like um you know it's it's it's obviously very kind of like some something that augments you right if you're calling something with Devon um um then it's very much kind of like you're, you know, the software engineer sitting next to you and they have their own virtual machine where all of that operates and and you know, they've spun up the repo themselves, right? Um, cloud code is kind of like handing the reins over to to your AI buddy to take the wheel of your computer, right? Which I think is is is an interesting paradigm, frankly. \>\> I agree. I think the the thing that struck me about it was it was a it was like a full send to a new version of a of agentic engineering where previous except for Devon um but previous iterations were always well the AI is like on the side and the CLI form factor was actually no no all you need is to talk to the AI and that was the first time um that that happened for something that you're using on your own computer, which I think is the other really crucial component, at least for me, because having it be able to run bash commands on your computer, makes it way more extendable and customizable than um if it's in some environment that you don't fully control. Uh or sometimes, for example, in codecs, the environment would be um spun up and then spun down. In Devon, I think it's I think it's more consistent, at least last time I checked, it was it was a consistent environment, but still harder to customize than your own machine. Yeah. Yeah. Yeah. And so, so I think there's a lot of different it's we talk about the entire kind of like there's like a hyperspace of all of the decisions that you can make, right? There's kind of like um you know synchronous versus asynchronous. There's like local versus remote in terms of like the environment that it operates in. There's like um you know in in the IDE versus in other things there's like single player versus multiplayer and whatever. Um and I think what we're seeing um is just there's there's obviously you know it's there's some exponential amount of possibilities in the space. I think a lot of different single points of of the space are pretty interesting and I you know I think Devon is the one point of the space. I think Windsorf you know and other ideas are another point of the space. I think I think Anthropic unlocked a new point in the space with claude code. I think that the um frankly I think that there will be a lot of these points that exist for quite a while and I think that the full suite kind of should have um should have a lot of these different experiences because obviously it depends a lot on your particular use case or your flow um which one is best at each point in time. \>\> What do you think the trade-offs are of the point in hyperspace that you're in? So in particular, I think the thing that makes Devon unique, at least in in my testing, has been it's an agent that lives on its own computer in the cloud persistently that you can talk to at any time. Uh which is just it is a different bet than pretty much any other big company has made. \>\> I mean it is almost like onboarding a software engineer, right? But but once you do there are a lot of tasks because Devon has its own environment and because it can go and learn how to test things and run all the tests itself um that it just can do you know that that that basically nothing else out there can do. Um, and I think it's interesting for us because, you know, even before the Swinsurf, you know, acquisition, we were already thinking about this question of, well, what should we do to um, basically, you know, make make it a lot easier, you know, make an experience that's a lot more accessible for folks, right? And we were talking about a few different ideas and, you know, more synchronous experiences. Um, and and then obviously, you know, everything happened with Winter. uh it was it was it was a great opportunity for us and and so a lot of how we see it today is is you know I I think there is going to be a lot of work in terms of not just like onboarding the agent but also in in human software engineers themselves learning how to work with more and more async agents \>\> and I I think it naturally has to kind of transition from a sync to an async environment um and so that's that's kind of how it's what has led to a lot of our thinking today is you know using windsurf or or having windfur as kind of like a a really fast time tovalue, you know, option that you can immediately just kind of download and use and get a lot of value of. Over time, you learn how to work with a, you know, cascade agent or you learn how to kind of like use the deep wiki indexing in Windsurf and and then naturally that takes you kind to to more of the ASIC flow. Um, but but yeah, \>\> that makes a lot of sense. I'm curious. One of the things you said earlier that stuck out to me and I think is really true is that there is um there's a tight dependency between model capability and what the right affordance or what the right harness is to use that model. Um so uh you know for example with cloud code it becomes possible to do a lot of the CLI stuff because Opus 4 and other models of that generation are good enough to to to make that work. Whereas for GBD3.5, if you did a CLI, it would have been horrible. Um, and I I think I I my question about that is you guys don't have your own models as far as I know. Maybe you have fine-tuned versions of models, but you're not building your own coding models. Is that true? \>\> So, we we do a lot of post-training of models. So, we do like fine-tuning and RL and things like that. We don't train we don't pre-train base models. \>\> Yeah, I guess why not? So given that there's you know given that there's a lot of overlap between the capability of the model and what the right harness or affordance is to use it um does that make you worried about competing with open AI or you know anthropic when when they have this like very tight coupling between these two things that are evolving together very rapidly. Um, on the other hand, I think post training very much is um, and you know, some of the examples that I'd give you are things like, well, you know, we want Devon to be able to predict its own confidence to have an opinion on, you know, how likely it is to be able to do this or how well it understands a task or things like that, right? Or, you know, maybe a more like direct practical one is like, all right, you know, one thing that engineers do a lot day-to-day is pulling up the data dog, finding the corresponding logs, and using that to debug what went wrong, and then making the right edits, right? And that's a very specific flow that you obviously need to have like custom training data in for you know the the models don't just learn that on their own right all of these things um they fit actually quite naturally into post- training um as a category um and and and so you know I think from our perspective it is it is really a question of what we think we spike most on and where we want to focus you know I think as a startup your your edge um it always has to be speed and focus and and I I think for us it's it's it's kind of like I I think we we know what our core DNA is about and and a lot of it is just like understanding the nuances of real world software engineering and basically teaching that to the models in a way that that that you can build a great product experience. Um and that's what we focus on \>\> in post- training world right now. RL environments are like really hot and it strikes me that you all have been probably purposely building the perfect RL environment. um for for post-training a software engineer. Tell me about that. \>\> Sure. Yeah. No, I mean I it's one of the beautiful things about code, right? And people, you know, people talk about this obviously, but but the fact that code is um um it has a much cleaner feedback loop because you could read you can run the code or you you know you have all the uh there there's all the version control. You could see every commit that was made in history. You could you know, you you have so many of these tools which you would love to have um in creating these things. Um, and I guess the only thing I would say here is, um, it really does come down to just building the exact custom environments for the use cases that you care about. Um, and so, so, so all this, you know, we just talked about like data dog or other there's there's honestly like hundreds of these, you know, within code and like, you know, random things that come like cobalt, you know, turns out there's still a bunch of cobalt out there in the world, right? Um, and it's not something that the language models are are are super kind of, you know, adapted to understandably, right? But but like that is that that that is real work and and you know, real stuff that that that takes a lot of time for people today, right? Um, and and I think a decent bit of the work of RL, you know, I I I've said this before, but but you know, my my high level view on RL is kind of like the platonic ideal of RL is that you can go and solve any benchmark, right? And um a and once you know once we have that which I think we're getting closer and closer to having that then the question is kind of just like okay well what's the benchmark right and now the question that I think a lot of these application layer companies are thinking about is basically what is the benchmark like what is the exact set of tasks and environments what are the tools that you're going to use what are the decisions you're going to make how are you going to decide whether is a success or failure um and if you have described all of those things exactly and you've kind of collected enough data points around that then you can train a model that just does it it's kind of a crazy. You know, it's it's it's kind of insane to think about that, but but obviously it it just means that to your point like the having the right environments and the right use cases is even more important. \>\> And I'll say that's very hard even like h getting that getting the answer like how do we decide whether this is good or or whatever is that is actually quite hard. I guess it strikes me though that there's there's maybe two two ways to solve the the kinds of problems we're talking about. Um so for example making a great software engineer one is um having an having RL environments set up that mirror the kinds of problems that uh a normal software engineer would uh would encounter and then uh using that to generate data to train the model to be able to solve those problems like the data dog example right um and and basically having a company enumerate what are the likely things that people are going to have to do and what are our users saying they want to collect all that data and then train the model. On the other end though, uh you know there's a lot of talk about continual learning uh which I think I kind of think continual learning is already happening. It's just very sample inefficient. Um so so on the other end of the spectrum instead of having to do all that work, you just if you just made the model um more sample efficient to be able to try stuff and and learn, it would it would make it so you wouldn't have to do as much of the uh you know RL environment post-raining type stuff. Um, so why why make the bet over here in the in the RL environment post- training side instead of on the more sample efficient learning side? \>\> Yeah. Yeah. Um, no, it's a good question. Uh, it's a it's a really good question. I I think like um I'll give you my high level view on on this is just kind of like a almost like a philosophical question even of like basically if we are going to have the full you know talked about all the reasons that ASI or AI is like a bad but but the the full you know intelligence that can do everything that we want it to do then obviously at some point it has to learn all of the practicalities of the real world right and I would argue we're bottlenecked by that right now we're not bottlenecked by pure logical reasoning like you know we could do some pretty insane logical reasoning with with language models Um, and so like how do you learn all the you know for an accountant who's doing all their work every day or for like a parallegal who's doing you know how do you learn the practicalities of what somebody does that way and and build a model that is like intelligent about that right and obviously you know the the best way to learn something is to do it and so so you know you need this actual data of of you know what does what what this particular parallegal does in some form right and and so then to your point I think there's kind of this fork in the road of where uh where that data comes from. And I would say I would add a third one or you know just just for completeness sake I think most people agree this but but but the the three that come to mind are one is well the data exists in the world already you just have to go get it. That was kind of like the pre-training view of the world, right? You just take more and more of the internet, you train it all, and because everybody has stuff on the internet. If you just keep doing that over and over, eventually you get a model that knows everything, right? Um, two is kind of like, well, it has to go be built by, you know, experts themselves. You have to go and like really curate an environment and figure out exactly, you know, like these like 500 environments of of your one task and then do that for every task that you could possibly imagine. And and that's how you you do that. And then three to your point is like you have an agent that can go out and do it by itself, right? Um so so I think two and three are kind of the that that's the those are the versions that you're describing of like do we go and handcraft the environments versus um versus is there some continual learning that works out, right? \>\> Yeah. Like it it does it and fails, you know? You you on board it to your company and it up the how to log into Data Dog a few times, but then it figures it out, you know? \>\> Yeah. Yeah. Um, and I think the um I think the short answer is basically I think we will get to three. Um, but I think a lot of the problems that you solve along the way actually naturally apply to both, right? And so so I I think it's kind of one like the pre-training world I, you know, personally I kind of think we've we've roughly, you know, we're kind of converging on on a lot of the capabilities on pre-training at this point. Um, two is is this RL world which is like actively the the the world that we're all in, you know, is going and doing custom RL to to to find particular capabilities. And then three is this kind of like long-term, you know, continual learning, which will obviously unlock some really big things. Um, but I guess I would just point out that for for both two and three to your point, a lot of what you have to do is you just have to create actual agents that can operate in the real world, right? And the way I kind of think about it is I think the most important thing in two that's making it more successful is you can just be much tighter about curating the reward function, right? Um and so like simple example of this is you know one of our evals or one of our environments um is is like um you know there's a there's a graphana dashboard that you need to set up and for some reason it's not working so please figure out what went wrong. And it's very much meant to be like a messy real world software. This is like the kind of stuff that you'll run into day-to-day. And the what the way the task works is obviously you go and install the packages. You run the code and stuff and then you find some error. It turns out the error is because you know the the version lock of the packages you know was like slightly off in a way that makes it you know this kind of stuff you run into all the time as an issue. So then you have to find this and you realize you got to downgrade the version of this and then you know that leads to the next thing and you figure out what went wrong with that error and then you get the thing running right and at the end there is a dashboard and the really beautiful thing about an eval like this is you can just make the eval just so what does the dashboard say and the thing is if you did not get the graphana dashboard running you will never be able to answer that question and if you did get it running you will always get the right answer right a and this curation just means you have a much much tighter feedback signal right for for for for three, um, you would want to be able to do this kind of like live in the real world and have that feedback cycle. The problem is it's like a lot tougher of like, well, you could get the dashboard running and then have the wrong number and then like how are we going to know that that was wrong, you know, or we could do this other thing, right? Um, a and so it's it's not an unsolvable problem. I I mean, I think I think we will make more and more progress on it over time, but I guess my point is just like either way, building the full environments and the tooling for the agents that can do this is kind of the primitive that you need to be able to to to do to do both of these routes. So \>\> that does that does make sense. Um the thing that comes to mind there is or or a worry which I'm I'm curious for your opinion on is the the graphana eval in that case how generalizable have you found the um being good at that eval is? So the fact that it can set up a graphana dashboard if Graphfana, you know, changes the way that their whole system works uh and that that breaks the eval. Yeah. Like are are you training is it does it end up being too brittle if it's trained for these specific kinds of environments that could end up changing like pretty fast? \>\> Yeah. So the first answer is yes, you do need like a bunch of environments naturally, right? Um but but but I guess the second answer, you know, the the meta level answer I'd give to you is is kind of like it really depends on how you set up the task, right? And so you know if the task is literally just like all right you just have to recall exactly what packages are needed for to to to to run this version of graphana from memory then yeah it's like that's not very generalizable because as soon as the next version comes out that's not going to be a null right but if the task is meant to be such that okay you're going to go and google this and then you're going to go and read the docs page of graphana and you're going to use that and understand what went wrong and then you're going to look at the error in your logs and find the file that that corresponded to you're going to read that file you're going to use that right so so it's I guess my point is like you We as humans all figure it out somehow. And and the the way I would kind of broadly describe that we figure it out is we actually inter interact with the real world in in a way that like gives us the information that we need rather than just like pulling it all from memory, right? And as long as your agent is set up to do that, that skill is something that is very generalizable. \>\> What's something in in AI that you're excited about that has nothing to do with cognition or Devon? \>\> Oh, interesting. I've always uh I've I've always felt like these personal agents should just be a thing. Like I I'm surprised it hasn't happened already. I I guess this is the way I would put it. Like you know I think obviously there's like operator, right? And there's kind of some deep research I think is another good example. But but but like I think just like a mass consumer agent that you can just have on your phone that just takes care of things for you. I mean it's it feels like the capabilities are there for that. Maybe I'm wrong, but and it feels like something like that would be so valuable, you know, like everyone always gives the example of like \>\> like schedule my dentist appointment or whatever. Like what what does it do? \>\> Yeah, exactly. It's like everyone always gives the example of booking flights for some reason, which is great obviously, don't get me wrong, but like this is not the only thing that I would have my personal agent do like that all all the like like I would love to have my personal agent go and call and book a reservation and then you know deal with various messages and then make sure like the package delivery like went through and then go and you know it feels like there should be something there or it's just like buying my Amazon packages for me just like hey like I need another order of X or Y or whatever it just goes against um Um a and I've always thought yeah that that should be something. It's funny because we actually I mean we we obviously build Devon like entirely for coding. Um but we've kind of messed around with it of just seeing like hey like can can devon do this you know uh and it's like yeah like we actually order all of our Amazon packages with Devon now you know and \>\> that's amazing. Well, the thing is it's again it's you know the Slack message or the linear ticket is like not the right form factor for that and somebody else should build the right you know but but it's it's it's like the Asian capabilities are there I guess is my point and and you know with Devon it's like because you have memory and because you you know are storing secret keys you might as well be able to handle like your Amazon account as well and because you have the browser like that is actually the set of things that that you need to be able to do that but but but it feels like something like that hasn't really like really taken off um in in a way that I would imagine, you know, I like I I I would imagine that 12 months from now we will not be, you know, like like that that that that we will have this, but but but I would love to have it sooner if possible. So, \>\> that's really interesting. Well, I hope to have you back on the show 12 months from now talking about uh the future of personal agents and hopefully some really cool updates to Devon. \>\> Yeah. Yeah. Awesome. Thank you so much for having me. Oh my gosh, folks. You absolutely, positively have to smash that like button and subscribe to AI and I. Why? Because this show is the epitome of awesomeness. It's like finding a treasure chest in your backyard, but instead of gold, it's filled with pure unadulterated knowledge bombs about chat GPT. Every episode is a roller coaster of emotions, insights, and laughter that will leave you on the edge of your seat, craving for more. It's not just a show, it's a journey into the future with Dan Shipper as the captain of the spaceship. So, do yourself a favor, hit like, smash subscribe, and strap in for the ride of your life. And now, without any further ado, let me just say, Dan, I'm absolutely hopelessly in love with you.