A talk by Lance Martin (Member of Technical Staff, Anthropic) on building agent harnesses for reliable and secure long-horizon work with Claude. Topics include decoupling the "brain and hands," self-verification, self-learning, and designing evolving agent harnesses. Speaker background includes work on the Claude Platform (Claude Managed Agents, the claude-api skill in Claude Code), earlier work at LangChain, and prior experience in self-driving car vision and a Stanford PhD. # Channel Presentation Observations — "Claude for Long-Horizon Tasks — Lance Martin, Anthropic" ## Video Context - **Format**: Recorded conference/workshop talk with live audience Q&A ("this is like my third year coming to this conference," "let me open up for questions"). - **Speaker self-identification**: The speaker is Lance Martin, speaking on behalf of Anthropic: "let me talk a little bit about… our view of async agents at Anthropic and some things we've been up to lately." - **Content type**: Technical product/engineering presentation covering Claude models, the Messages API, Agent SDK, Managed Agents, and Claude Tag, plus empirical experiments (Parameter Golf, Claude Plays Pokémon, Continual Learning Bench). ## Observable Intended Audience - **Technical AI practitioners / developers** building with Claude APIs — evidenced by API-level discussion ("Messages API… Agent SDK… Managed Agents") and mentions of "engineering blogs" and open-source benchmarks. - **Conference attendees already familiar with agent concepts** — assumes familiarity with terms like "compaction," "prompt injection," "recursive language models," "the loops trend," and "bitter lesson." - **Practitioners thinking about agent architecture** — the talk is structured around architectural guidance themes (decoupling brain/hands, verifiers, memory, org-level harnesses). ## Claimed / Demonstrated Scope - **Official Anthropic perspective**: Explicitly framed as "our view of async agents at Anthropic" — a first-person institutional framing, not a personal opinion channel. - **Product announcements and roadmap commentary**: Discusses released products (Managed Agents, Agent SDK, Claude Tag) and predicts trends: "we're going to see the rise of kind of harnesses that operate across orgs" and "asynchronous agents are going to be increasingly proactive." - **Hands-on experimentation demonstrated**: The speaker describes running his own benchmarks and experiments — e.g., "I set up a kind of a verifier loop using manage agents and outcomes to test the ability for Opus 4.7… on this task" (Parameter Golf), and "I played a lot of games of Pokémon with Claude to find this." ## Self-Description vs. Demonstrated Presentation - **Explicit self-description**: Positioned as an Anthropic insider involved in product efforts ("I was involved in some efforts to build kind of async agents"; "we released Agent SDK"). - **Demonstrated presentation style**: Practitioner-researcher tone combining: - Architecture diagrams and benchmark results ("So, I'm plotting here different Claude models and their task horizon"). - Analogies for technical concepts (Claude as "a light source," products as "windows"; hippocampus/cortex memory analogy for in-band memory vs. "dreaming"). - Humble, self-correcting phrasing in Q&A — e.g., "why is it that non-frontier models are not kind of in that regime? I am actually not necessarily sure." - **Opinion offered with hedging**: In Q&A, gives design guidance as observations, not decrees: "I don't necessarily think that it has to be the case that you use a file system for memory." ## Q&A Session Evidence - Audience questions cover frontier-vs-non-frontier model gaps, memory substrates (file systems vs. databases), and verifying "dreaming" correctness — indicating an **expert audience** engaging at a research/engineering depth. - Speaker attributes claims to evidence where possible: "we've actually run a lot of different evals showing that dreaming can indeed improve performance," while acknowledging limits: "evals are important in like your own context." ## Notable Presentation Patterns - **Data-backed claims**: Multiple references to empirical measurement ("This is measured by meter"; "I saw this in five replicates. Five out of five replicates"). - **Transparency about uncertainty**: Distinguishes anecdote from rigorous findings ("this is a fun anecdotal example from Pokémon"). - **Named external benchmarks/products**: References OpenAI's Parameter Golf benchmark, Codex results, and "meter" — indicating cross-lab awareness rather than purely promotional framing. # Delivery Observations — Claude for Long-Horizon Tasks (Lance Martin) ## Overall Tone and Register - **Conversational, workshop-style delivery**: casual openers and asides like > "Good to go. All right. Well, take a quick sip and then let's start." - Uses **informal vocabulary** throughout: "spooky," "pretty crappy," "a little bit spooky," "quite bad," "pigeonhole." - Self-deprecating humor appears repeatedly, e.g., > "maybe this talk, maybe not this talk" and > "I have actually created a lot of Slack bots myself... In fact, I've created many Slack bots that are quite bad." ## Recurring Phrases and Verbal Habits - **"That's the key point" / "the key idea"** is his most frequent signposting phrase, appearing many times: - > "That's the key idea." - > "So, that's the key point." - > "That's the key intuition." - > "That's really the key point here." - > "That's the key differentiation." - > "That's the key observation." - **"Kind of"** as a filler/softener appears extremely frequently ("kind of a shift," "kind of interesting," "kind of tricky"). - **"So,"** as a sentence-starter is pervasive ("So, theme one is...", "So, the story here is...", "So, this is a kind of a fun and interesting challenge..."). - **"Um"** appears throughout, including at times in clusters. ## Teaching Habits and Structure - **Numbered themes**: explicitly structures the talk as "theme one... the second theme... theme four," signaling organization. - **Analogies from everyday life**: - Models as light: > "you can think about Claude as a light source and you can think about products as windows that allow the light to pass through." - Human memory/dreaming analogy for agent memory: > "the hippocampus is kind of writing traces of... short-term... memory. When you go to bed at night, though... dreams." - **Repeated framing of "in-band vs. out-of-band"** processes (in-band memory writing vs. dreaming) — a stable conceptual contrast used across several slides. - **Narrative anecdotes as evidence**: "Now let me tell you a story about how I've used this" and the Pokémon trapdoor example ("this was a very hard one lesson, so I hope you appreciate it"). ## Signaling Confidence and Uncertainty - Explicitly states uncertainty when asked why non-frontier models lag: - > "why is it that non-frontier models are not kind of in that regime? I am actually not necessarily sure." - Elsewhere qualifies claims as personal observation: > "the main differentiation... is kind of this distillation step... That's kind of the key difference that I found." - Frames one example as anecdotal: > "this is kind of a fun anecdotal example from Pokémon." ## Restating Questions Before Answering A consistent Q&A habit of paraphrasing audience questions: - > "So, the question was kind of on the the gap between the frontier models..." - > "Um the question was about um kind of like the the best memory substrate..." - > "Basically, the question is so, you do this dreaming thing..." ## Audience Awareness and Interjections - Occasional audience sounds (transcribed as "[snorts]") punctuate the talk; speaker does not visibly address them in the transcript. - Closes with a standard handoff: > "let me let me just open up for questions and um, thank you for listening." ## Notable One-Off Features - **Stuttered restarts** appear in moments of emphasis: > "And and and and effectively," > "it it it is non-destructive." - Slowdown for a subtle point: > "Um this is kind of a subtle point that actually um I want to think about carefully." ## Summary Stable patterns: heavy signposting with "key point/idea" phrases, "kind of"/"so"/"um" fillers, analogy-driven teaching, anecdotal demos, question-restating in Q&A, and explicit hedging when unsure. Isolated moments (stutters, direct audience interjections) do not appear to form broader patterns on this transcript alone. # Observation Record: "Claude for Long-Horizon Tasks" — Lance Martin, Anthropic ## Framing Metaphor: Models as Light, Products as Windows - Martin frames **Claude as a light source** and products as **windows** through which the light passes; the window needed to "see" the model shifts as capabilities grow. - **Task horizon chart**: In the "Opus 3 days" (~2024), models could do roughly **10–20 minutes of autonomous work** (measured by "meter"). That regime suited **autocomplete and chat** — products with the human heavily in the loop. - Past year: rise of **synchronous coding agents** like Claude Code, with models doing maybe **an hour of work**, typically run locally so users could steer them. - Martin recalls efforts (some he was involved in) to build async agents during the one-hour-horizon regime; **async "as an experience is kind of bad"** at that horizon — the model goes off, hits an error, and returns quickly. - Claim: unlocking async required **longer task horizons**, which are now emerging. ## API Surface Evolution - **Messages API** (~2 years old): prompt-response, no deployment concept; users roll their own harness and deploy it to build agents. - **Agent SDK**: released during the coding-agent wave; a programmatic way to call Claude Code — "basically us giving you a harness." - **Managed Agents** (released in "the past few months, since April"): packages both the harness and managed deployment infrastructure. ## Theme 1: Decouple the Brain from the Hands - Initial Managed Agents design put harness and sandbox in the same container; problem: if either dies, **the session is lost** — bad for long-horizon agents. - Additional concern: placing **credentials in the same container** with a highly capable agent running ~10 hours unwatched "can be a little bit spooky" — a stated security concern "especially as models get extremely capable." - Solution architecture: - **Brain** = stateless harness process talking to a session. - **Session** = append-only event log. - **Hands** = containers/sandboxes doing execution. - Claims: Claude can manage **many hands** (multiple containers) easily and effectively; if harness or sandbox dies, the session is backed up in the log; credentials live in a **separate vault**, never in the sandbox. - Connects to **recursive language models** work: the session becomes an external context object the model interrogates. Benefits for context engineering — the context object is **persistent, append-only, non-destructive**; unlike naive compaction (which discards non-compacted context), the model can always fetch old context. ## Theme 2: Use Verifiers - Problem: when one context both does work and grades it, you get "odd artifacts and confabulation"; models are often "not properly tuned to do critical verification." - Solution: **separate verification into a separate context window** — a general trend Anthropic has discussed in engineering blogs; the verifier context can be tuned specifically for critical verification. - **Build/verify loops**: build agent + verifier agent; verifier holds a goal or rubric and checks the build agent's work; the loop exits only when the independent verifier confirms the desired outcomes. Primitives: **goals** (Claude Code) and **outcomes** (Managed Agents) — same principle: measurable end state + independent grading context. - **Case study — "parameter golf"** (an OpenAI benchmark): train a small model on 8×(8100?) GPUs in under 10 minutes; Y axis ~loss, lower better. Martin ran a verifier loop with Managed Agents and outcomes testing Opus 4.7 and a "mythos class" frontier model. The loop ran until his outcome was satisfied (exactly 20 iterations meeting benchmark criteria). - Claim: frontier-capability models are "extremely good" with loops; steering signal is encoded **into the environment** rather than the human, letting the model self-correct from verifier feedback. - Key point: **loops + high-capacity models = a very good general primitive for long-running async work.** ## Theme 3: Self-Learning / Memory ### Human analogy - Hippocampus writes fast, short-term experiential traces during the day; **dreaming** is an offline/out-of-band process that consolidates important details to long-term memory in the cortex. ### In-band memory writing - Memory tools = ability to write to a **memory directory** (file system), "that's really it." - **Claude Plays Pokémon, Sonnet 3.5**: with memory access, "in-band" writing is "not very good" — memories are "pretty crappy," tactical not strategic; limited game progress. More recent models (he references "4 6") write much more strategic notes with much further progress. - **Continual Learning Bench** (open-source; Martin ran it): a sequential QA task over a SQL database with memory writing between steps; performance improves across model generations. - Claim: models get **natively better at in-band memory writing** with capability. - Distinction found: higher-capacity models do a better **distillation step** — they save abstractions that generalize to future sessions rather than isolated facts. ### Dreaming (out-of-band memory correction) - Nuance: in-band memory can be **incorrect or locally-but-not-globally optimal** (optimized for the current task, not future ones). - **Pokémon anecdote** (Martin emphasizes it took many games to find): Claude wrote an incorrect memory about a location; this caused consistent mislocalization and a fall through a trapdoor — **5/5 replicates** with raw memory store fell into the trap. With dreaming, the error was corrected and the agent avoided the trap. - Visualization: no-memory baseline (gray) barely progresses; memory (orange) repeatedly backtracks via the trapdoor; dreaming traces consistently fix the memory error and advance. - Key intuition: **memory errors persist unless an offline process corrects them**; dreaming reviews memory stores and prior traces to find and fix errors. ## Theme 4: Org-Level Harnesses (Claude Tag) - Reaction to Claude Tag was dismissive ("Ah, Slack bot"); Martin acknowledges he's built "many Slack bots that are quite bad," but argues the interesting part is **not Slack access** — it's the rich system underneath: an **org-level, multiplayer harness**. - Properties: own identity and credentials not tied to a user; access to **organizational-level context**, not just personal local context. - Claimed benefits: checking others' work before experiments, deduplicating findings, internal research, and giving everyone a well-developed harness on **day one** (vs. weeks or months for new employees to configure personal harnesses). - Prediction: rise of **harnesses operating across orgs and many users**, on longer async time frames. - Async agents will be increasingly **proactive** (vs. reactive local agents): e.g., Claude Tag configured to alert users about org-level context. - **Multiplayer steering** — a single harness steered concurrently by many people — is an important UX shift. ## Q&A Positions ### Why do frontier models outperform on long-horizon tasks, and how long will the gap last? - Martin notes frontier models (citing "Codex like 56" and "mythos class" models) are in a **12+ hour regime on meter**; he is "not necessarily sure" why non-frontier models lag. - His view: long-horizon agency requires more than model capability — **memory** (encoding productivity preferences, knowing when to reach out when stuck), **security** (prompt-injection resistance), and **agent architecture** (brain/hand decoupling for security and failure resistance) must come together. Frontier labs invest in all of these, which "might be why" a product gap has emerged. ### Best memory substrate: file system vs. database? - Martin: it doesn't have to be a file system; a database "could work fine." - Key differentiation: what **doesn't work** is explicitly prescribing memory structure (a memory schema with predefined memory types) — "not very bitter lesson pill." Models manage their own memory better than you can intuit types ahead of time. - Key point: **general, freely-manipulable substrates good; prescribed memory schemas bad** — models can reason about their own memory/context structure better than you can prescribe it. ### How do you know dreaming corrections are correct? - **Evaluations** are one way; the Pokémon example is "fun" and "anecdotal." - Claim: many evals have been run showing dreaming improves performance; still, users should run their own evals to confirm the offline compute is worth it. ## Qualifications & Tensions Preserved - Martin explicitly flags uncertainty on *why* non-frontier models lag on long horizons ("I am actually not necessarily sure"). - The Pokémon dreaming result is presented as anecdotal, supported by 5/5 replicates on one trapdoor error, with evals recommended for the listener's own context. - The transcript's GPU specification ("8100 GPUs" / "eight 8100 GPUs") is garbled in the source; recorded as expressed, without correction. # Speaker & Provenance Observation Record ## Video - **Title:** Claude for Long-Horizon Tasks — Lance Martin, Anthropic - **Video ID:** `01a15ca8-69dd-78c6-aac0-04f9e07e090d` ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Lance Martin** | **Creator / primary presenter** (Anthropic) | Title attributes video to Lance Martin, Anthropic; delivers the full talk in first person ("our view of async agents at Anthropic"), references his own experiments ("I've used this", "I ran called Continual Learning Bench") | | **Unnamed audience members** | **Other — Q&A questioners** | Ask questions after the talk (e.g., on frontier vs. non-frontier model performance gaps, memory substrates, correctness of dreaming) | | **No narrator or guest** | — | Transcript contains no narration track or guest interview | > Note: Q&A questions are partially inaudible/paraphrased by the speaker ("So, the question was..."), so questioners are not identified by name. ## Referenced Sources & Third-Party Claims - **OpenAI's "Parameter Golf" benchmark** — described by speaker as a challenge "put up by Open AI" testing ML research (training a small model on 8×8100 GPUs in under 10 minutes). Speaker reports his own verifier-loop results with Opus 4.7 and a "mythos class" frontier model. - **"Meter" benchmark** — cited for measuring autonomous task horizon; speaker cites results including "Codex like 56... in that 12 plus hour regime" (attribution to third-party benchmark results). - **Continual Learning Bench** — described by speaker as "an open-source benchmark" he ran, using a sequential question-answering task with a SQL database. - **Claude Plays Pokémon** — referenced work showing memory-writing capability (Sonnet 3.5 vs. newer models). - **"Recursive language models" work** — speaker references it generally: "you guys may have kind of seen or come across the recursive language models work"; no specific citation given. - **Anthropic engineering blogs** — speaker says the separate-verifier-context pattern "we talked about it in a number of different engineering blogs"; no specific links. - **Bitter lesson** — referenced conceptually ("bitter lesson pill") regarding letting models manage their own memory; no citation. ## Anthropic Products / Claims by Speaker - **Claude models** (Opus 3, Sonnet 3.5, Opus 4.7, "mythos class" frontier models) — capability claims across generations. - **Messages API, Agent SDK, Managed Agents** — described as Anthropic API surface evolution. - **Claude Code** — synchronous coding agent; "goal" primitive. - **Claude Tag** — org-level harness via Slack; speaker defends it against "Slack bot" criticism. ## Promotional Segments - **None flagged as explicit promotion** (no sponsor reads, ads, or product plugs for third parties). However, the talk itself is an **Anthropic product presentation** promoting Managed Agents, Claude Code, and Claude Tag — organizational advocacy by the speaker, though presented as a conference workshop talk. ## Availability & Uncertainty - **Comments:** No metadata supplied; comment availability **unknown**. - **Curated metadata:** None provided beyond the title; speaker identity is established via the title line only. - **Inaudible portions:** Multiple audience questions are only reconstructed via the speaker's paraphrase; the exact wording of questions is not preserved in the transcript. - **Model names:** Some model references in the transcript appear garbled (e.g., "mythos class," "Opus 4.7," "4 6," "Claude Tag" / "Cloud Tag") — recorded as transcribed, without correction. \>\> Good to go. All right. Well, take a quick sip and then let's start. It is great to be here. Um this is like like my third year coming to this conference and I always really enjoy it. And thank you for coming to this workshop. I know there's many interesting talks. Let me talk a little bit about um our view of async agents at Anthropic and some things we've been up to lately. So, this is kind of a way I think about models in product. So, you can think about Claude as a light source and you can think about products as windows that allow the light to pass through. And what's kind of interesting is over time, the window that you need to actually kind of see the light of the model kind of shifts. And we've seen this over the past few years. So, I'm plotting here different Claude models and their task horizon. So, how much autonomous work can they do over time? And you might recall back in like the Opus 3 days, this was kind of like 2024, models could only do, you know, maybe 10 to 20 minutes of autonomous work. This is measured by meter. And in that regime, only certain product surfaces made sense. Things like autocomplete, things like chat, where your human is very in the loop cuz the model's really only doing a very short amount of work before you're steering it. Now, the past year we saw the rise of synchronous coding agents like Claude code. And this is, you know, a kind of a shift because then models could do maybe an hour of work. So, it made sense to have them run, but typically locally, where you could still steer them easily. And it's kind of interesting because during this regime, I remember efforts and I was involved in some efforts to build kind of async agents. But when models can only do like an hour of work, async as an experience is kind of bad. Um the model goes off and it like hits an error and it comes back to you over a short period of time. In order to really unlock async, we needed longer task horizons. And so we're starting to see that now. And kind of with this shift in capability and time horizon came a shift in the API surfaces. So if you look at the lower left, Messages API came out like 2 years ago. It's basically prompt response. It's great for building harnesses, but it's a very simple API. Again, you're just passing in a message and you get a response out. There's no sense of deployment with that. So you basically take Messages API and you can roll your own harness, you can deploy that harness, and you have an agent. Now over the past year, we saw the rise of, you know, coding agents in particular. So we released Agent SDK. So that's basically a way to programmatically call Claude code. And that's like basically us giving you a harness. But over the past few months, since April, as we've seen longer longer task horizons, we released a new API called Managed Agents, which basically packages both the harness as well as all the managed deployment infrastructure for you. And I'll talk about some of the themes that underpin this new uh surface, Claude Managed Agents, and and some of the themes that kind of extend beyond just Managed Agents broadly to think about this kind of new type of asynchronous agents, uh which can apply of course to Claude and other types of kind of longer running long horizon agents. So theme one is decoupling the brain from the hands. Um so when we first set out to build Managed Agents, we started with the container. We put the harness in the sandbox in the same container. Now the problem here is, what happens if the harness dies or the container dies? What we saw is you actually lose the session. So basically, this architecture is kind of tricky for long horizon agents because what can happen is your agent's running, and And container dies, you lose everything with it. Also, as models get more capable, putting the credentials in the same container with the agent itself can be problematic. So, for example, giving Claude access to a bunch of your secrets and letting it run for 10 hours and not watching it can be a little bit spooky and have some security concerns, especially as models get extremely capable. So, for this reason, we kind of decouple what we call the brain, that's the harness, from the hands, the execution environments, and manage agents to set up like this. So, the story here is that the harness becomes a stateless process that talks to a session. The session is an append-only event log and that can reach out to hands, which are just containers. So, that's just Those are sandboxes where work is done. And one thing that's interesting is Claude is increasingly capable of managing many hands. So, that is you can give one harness access to many different containers to perform to execution. And Claude can manage this very easily and and and and effectively. If the session, uh sorry, if the harness dies or sandbox dies, it's completely fine because the session is always backed up in this append-only log and credentials are never actually added to the sandbox. They're stored in a separate vault. So, this decoupling actually makes it quite reliable and safe, particularly for long-horizon tasks. And this is kind of one of the core ideas that underpins manage agents architecturally. And I think an interesting thing that falls out of this is related to you guys may have kind of seen or come across the recursive language models work. The session becomes an external context object that the model interrogates. And this has all sorts of benefits for context management. So, you think about it, when you're doing something like compaction, you're choosing some logic to retain some amount of context, and naively in a typical in a kind of a typical step, you're discarding all the context that you didn't compact. In this architecture, and also more broadly with recursive language models, the idea is that the context object is persistent and is unadulterated. So, it's append-only. And the model can always go back and fetch old context. So, basically creates a very very nice architecture for context engineering because the core context object is immutable in the sense that it's it's it's non-destructive and you can only and you only append to it over time. So, we've seen this be to be quite nice in terms of long horizon context engineering as well. So, the second theme is use verifiers. And one of the problems that we've seen with Claude and other models in general is that when you ask them to do a bunch of work and then say, "Okay, grade your work." If that same context is being used to both do the work and grade, you can get lots of odd artifacts and confabulation and and basically odd behavior. For example, this is just an image showing you can think about that that context window is filled with lots of different information and the model is grading itself, often it's not properly tuned to do kind of critical verification. And so, what we found is it's quite effective to separate verification into a separate context window. This is a very general trend. We talked about it in a number of different engineering blogs. Um and the reason is the verifier context can be tuned very specifically for the critical verification task. \>\> \[snorts\] \>\> And so, the way this works in practice is when you build loops, you can have a loop of a build context and a verifier context. And this can be a build agent verifier agent. And what happens is the verifier has some goal or rubric and it's verifying the result or work of the build agent. And this continues in a loop until verification is complete. And this is really the big idea behind this whole loops trend that you might have heard about and we found it to be a kind of a very powerful paradigm especially for working with some of the higher capacity models. And here are some of the primitives. So in Claude code you have goal and manage agents you have outcomes and the principles are really the same. You're setting up a measurable end state in both cases. You're using an independent context model. You're using independent context to grade over the course of this loop. The loop can run and you only exit the loop until this independent verifier has verified that it has the outcomes or outputs that you want. That's the key idea. Now let me tell you a story about how I've used this. So this is a kind of a fun and interesting challenge called parameter golf. It's a benchmark that set up that was put up by Open AI. And it tests models ability to effectively do kind of ML research. So it asks the model to basically take a small um kind of model and train it in with eight with eight 8100 GPUs in less than 10 minutes. And what you see on the Y is basically you can think about it as loss. So lower is better, okay? And what I did was I set up a kind of a verifier loop using manage agents and outcomes to test the ability for Opus 4.7 and one of our frontier models like mythos class models on this task. And what you see is basically I allow the model to continue to iterate until the outcome that I specify is is satisfied which is it finished exactly 20 iterations and kind of it kind of met all the experimental criteria as defined by the benchmark. What you see is the frontier capability models are extremely good with this pattern of kind of loops in software and and kind of verification because what happens is instead of encoding steering me and into like me as the human, you're encoding the signal into the environment. So, the model can self-correct when it receives feedback from, for example, the verifier. And using this kind of paradigm with very high-capacity models, you can get very strong results. So, the main point I'm trying to make here is that this paradigm of loops, which a lot of people been talking about today, paired with very capacity models is a very good general primitive for long-running asynchronous work. That's really the key point here. \>\> \[snorts\] \>\> Now, let me talk another about another theme of self-learning. So, the human brain has two kind of interesting systems for memory. So, one is as you go about your day, the hippocampus is kind of writing traces of kind of short-term, very fast kind of experiential memory. Like, you might remember what you had for lunch today. You had lunch an hour ago, you kind of remember, that's kind of written to short-term memory. When you go to bed at night, though, an offline process or out-of-band process, dreams. And dreaming stores certain important details to long-term memory in the cortex. So, for example, tomorrow, you might not remember what you ate for lunch today. That's kind of a local trace. But, like, if you had a very important experience today, maybe this talk, maybe not this talk, but if you had an interesting experience today, that might be written to long-term memory. That's kind of the point. These two subsystems in human work like this. And we've actually found memory systems with Claude actually can employ these same two principles. So, this is showing Claude's capacity as an in-band memory writer. So, you basically give Claude memory tools. And when I say memory tools, I mean it's basically the ability to write to a file system, that is basically a memory directory. That's really it. Now, this is showing some work on Claude plays Pokémon with Claude's on it 3.5. And here's the key point. When Sonnet 3.5 is given this access to a memory directory, and it can write memory, \{quote\} in-band, as it progresses through this game, it's not very good. So, the memories it writes are pretty crappy. It's kind of um it's kind of uh tactical notes. It's it's not very strategic. And the game progress is quite limited. But with more recent models like this is looking at 4 6, the notes are much more strategic and game progress is much further. So, the key point I'm making here is that Claude has gotten much better at this in-band memory writing across model generations. And this is another way to show that same result. So, this is a benchmark that I ran called Continual Learning Bench. It's an open-source benchmark. I took one of the tasks. This is a task that basically asked the model to perform a sequential question answering with a SQL database and it can write memory in between each step. And what you see is basically the performance improves across models. Um and so what this is kind of showing is that models get natively better at this in-band memory writing with respect to model capability. \>\> \[snorts\] \>\> And some of the most interesting things I found from this are that the main differentiation between There we go. Um the main differentiation between like a lower capacity model and a high capacity model is kind of this distillation step. And so, basically, higher capacity models have a better sense of like what abstraction to save to memory that'll be useful later. Like they're not just writing a specific fact. They're writing how do I how does this generalize to future sessions? That's kind of the key difference that I found that higher capacity models kind of have when they're writing memory. So, this is a very important thing to keep in mind that models are getting better better and better at this kind of in-band memory writing across model generations. Now, there's a little trick here which is very important. So, we talked about kind of in-band memory and we talked about dreaming. So, at night I dream and I write things to like long-term memory. Dreaming is very important because when I'm writing memory in band over the course of a day over the course of a session, sometimes you can write incorrect memories. And or you're writing things that are locally optimal, but not globally optimal. So, you're writing over the course of a task to like kind of help you solve that task, but not necessarily looking forward to future tasks. This is a very important nuance in this process of dreaming is is kind of an offline or out of band process that we've used to consolidate and improve memory. And I want to show you a fun example that I've used dreaming for. So, this is again Pokémon. And this is I played a lot of games of Pokémon with Claude to find this. So, this was a very hard one lesson, so I hope you appreciate it. Okay, here's the point. So, basically what happened is Claude wrote an incorrect memory, okay? And what happened is this incorrect memory was related to the location of The details don't necessarily matter. Uh the point is that this incorrect memory causes Claude to mislocalize itself or Pokémon to mislocalize itself, and it falls through this trapdoor, okay? That's the key point. So, it writes this incorrect memory. This incorrect memory causes it to mislocalize in the game, and it falls down this trap. This is very consistent. So, I saw this in five replicates. Five out of five replicates with raw memory store fell down this trap. With the dreaming, this error is corrected, and it's able to properly localize itself and not fall fall down this trap. And I'll show you kind of a fun visualization of this. This is looking at kind of memory traces or basically traces of game progress. So, going upwards on the Y axis is improvement. That's like moving to the next level. Going down is backtracking. So, what's interesting here, the no memory baseline, which is that gray bar, kind of doesn't make much progress at all. It just kind of like is stuck. It's like a particularly hard level, okay? The memory, which is the orange, actually keeps falling down this trapdoor and falls back. So, it backtracks. The dreaming traces though consistently kind of fix this error in its memory and proceed to the next level. So, this is a very practical example of how dreaming can kind of work out of band on your memory store to fix corrections. Because what it does is it looks at your memory store and it looks at all your prior traces or sessions and kind of can find and correct errors. That's the key point and that's why the dreaming process can be very helpful because in band, while Claude is writing to memory, it can make mistakes. And those mistakes get stuck in memory unless you have an offline process to kind of correct them. That's the key intuition. Um and theme four, and I'll open up for questions after this, is what I think um is kind of this trend that we're going to see moving towards org-level harnesses with async agents. And so, we released Claude Tag and a lot of the reaction was like, "Ah, Slack bot." And like, look, I have actually created a lot of Slack bots myself. I understand not every Slack bot is particularly interesting or great. In fact, I've created many Slack bots that are quite bad. But what's interesting about Claude Tag is not the fact that it's accessible through Slack. What's interesting about it is the fact that it has a very very rich kind of system underneath it, which I want to just cut touch on briefly. And in particular, what's interesting about it is it represents what I consider an org-level harness. So, agents historically have been kind of single player. So, you have an an agent like Claude code on your machine with your local context that you've tuned and configured for yourself. What's interesting about Claude Tag is it is a harness that everyone in the organization has access to and can use. So, it is a multiplayer harness. And what's nice about that is it has its own identity. Its identity and credentials are not tied to a given user and has access to organizational level context, not just my local context. This has many interesting and useful implications. Including the ability to like check others work before you do an experiment, the ability to kind of deduplicate findings, the ability to like do internal research, the ability to give everyone access to kind of a a very well developed harness on day one, whereas when you your own personal harness, often new employees takes them weeks or maybe even months to kind of ramp up fully to configure all the right connectors and so forth. So, org level harnesses are real leveler of the playing field and I think it was people kind of saw the Slack bot piece, but they didn't really appreciate the depth of kind of that the depth of benefit you get from building out org level harnesses. So, I do think that was kind of important thing to note. And I think we're going to see the rise of kind of harnesses that operate across orgs, across many different users that can operate increasingly on longer async async um async agents that can operate on longer time frames. That's kind of one clear kind of follow-up that we're I think we're going to see from this. And another thing where I think we're going to see is that asynchronous agents um are going to be increasingly proactive. Uh so, typically with for example like local locally scoped agents, they tend to be reactive. They're responsive to how you steer it. versus async agents increasingly have the ability to steer proactivity. And that's one very nice thing about Cloud Tag. Um where basically you can configure it to tell you things when like looking at this org level context, alert me with things I might need to know about. Um and this is a very important kind of new kind of UX that I think is going to be more and more common with async agents that kind of access to organizational context. And of course multiplayer. So, the the ability for a single harness to be steered by many many different people kind of concurrently is an important shift in agent UX uh that I think uh will be quite interesting going forward. So, um, yeah, let me let me just open up for questions and um, thank you for listening. \>\> \[applause\] \>\> Sure. \>\> Uh, one thing that we see sort of empirically and also in the benchmarks is that the frontier models perform better on these long horizon tasks than on the ones that have stacking. What they need to keep in mind that this is not What is your view on and like the guidance on it and like the right away model? \>\> Yeah. \>\> Um, what's your view on why that is and then how long the frontier will be able to maintain that gap? \>\> I see. So, the question was kind of on the the gap between the frontier models kind of on for example, a benchmark like meter on like long horizon tasks. \>\> Yeah. \>\> Okay. Um, so, in the latest results that I saw from like for example, Codex like 56, I think I'd also kind of is in that 12 plus hour regime on meter. So, I think you're right that like the frontier models like the, you know, mythos class models, strong models from Open AI kind of are in this like 12 plus hour regime. Um, why is it that non-frontier models are not kind of in that regime? I am actually not necessarily sure. I do think I do think um, in order to build agents that can effectively operate in this regime, it's important to note that it's not just the model capability. Like for example, with a Claude tag product, it's actually a combination of improvement in memory because memory is very important. If you have agents working for for example, 12 hours, you want to make sure that your productivity preferences are well encoded in memory. So, it knows when to reach out to you if it gets stuck for example. So, memory is very important, security is very important, so resistance to prompt injection. Um also like model architecture, like kind of the agent architecture is very important, that decoupling of brain and hand, so it's secure and safe and like resistant to failure. So, actually I think to build real agents that can operate in these long time horizons, a bunch of things need to come together in terms of like architecture, infrastructure, security, memory. And that might be why and and but Frontier Labs invested in all these areas, so that might be why it's become a gap in terms of like the agent products that we've released. And so we spent a lot of time, for example, building managed agents to kind of have these kind of considerations baked in. Yeah. Sure. Yeah, okay, this is interesting. Um the question was about um kind of like the the best memory substrate, so like why file systems versus, for example, databases. Um this is kind of a subtle point that actually um I want to think about carefully. So, I don't necessarily think that it has to be the case that you use a file system for memory. I think what's quite important that we've seen is that it's you want something that is highly programmable with simple primitives that the model can manipulate to like write, manage its own memory. So, for example, a database could work fine relative to the file system. But what I've seen doesn't work is when you specify the structure of memory for the model very explicitly, whether that's in a file system or database or whatever. Like a memory schema, I pre I kind of pre populate, here's the types of memories you need to save. Cuz I think that ends up being not very bitter lesson pill in the sense that models can learn to manage their own memory much better than you can intuit these memory types for the model ahead of time. So, I think what we've seen is that very general substrates for memory, be it just your database or file system, are good because the model can manage them freely versus a very very kind of like prescriptive memory schema they are trying to pigeonhole the model into. That's when you see performance drop. That's the key differentiation. Right. That's the key point. Let the model structure and maintain its own memory. Don't give it a prescribed memory schema. And that's like a common failure because models are getting good enough that they can manage their own memory much more effectively than you can reason about types of This is like very classically bitter lesson pill, but like you can Models can reason about their own memory and context structure much better than you can prescribe for them a way to structure their own memories. That's the key observation. Yep. Yes. That's right. Exactly. General substrates for for memory management. Yep. Yes. Okay. So, that's a good point. Basically, the question is so, you do this dreaming thing. You look at the sessions, you look at the memory store, you update the memory store. How do you know those are correct? Um so, evaluations obviously are one way to do it. Um this is kind of a fun anecdotal example from Pokémon showing that like you can do perform corrections via dreaming. The key point is that we've actually run a lot of different evals showing that dreaming can indeed improve performance for very intuitive reasons as you see here. But of course, evals are important in like your own context to confirm it's actually worth the offline compute. Yep. I guess we're done. Thank you all. \>\> \[music\]