Stefania Druga (Sakana.ai) presents an experiment on memory harnesses for long-running research agents, framing memory as a write/manage/read control loop rather than an attached database. Holding the model fixed, she compares recall policies on a ladder — no recall, vector RAG, a ranked decisions ledger, and an oracle given correct memory — across 68 xbench questions. Key findings: when a literature review task fits entirely in the context window, adding memory only adds cost with no accuracy gain (a boundary-marking negative result); on long-horizon tasks where the needed answer falls far outside the window, the ranked ledger recall policy performs best, even beating the oracle, since providing the right memory does not ensure the model uses it. Ranked recall was also cheaper, supporting the takeaway that bad memory is expensive and recall policy should be treated as a first-class metric. She also discusses context rot on long-horizon tasks, running local models on an M3 Ultra machine, and broader topics including the memory landscape and sovereign AI at Sakana. # Channel Identity Observations — Stefania Druga, Sakana.ai ## Explicit Self-Description (Creator Claims) ### Speaker and Affiliation - **Claim**: Speaker identifies as **"Stefania Druga, a research scientist at Sakana AI in Tokyo."** (transcript opening) - **Affiliation claim**: "This sovereign capability is part of a bigger ecosystem that is very important for us at Sakana AI in Japan. Um, we believe in the importance of sovereign AI today more than ever." - **Recruitment claim**: "We are also hiring. So, if you're interested... come talk to me." ### Stated Topic Scope - **Claim**: "Today I'm going to talk to you about memory harnesses for long-running research agents on device." ## Observable Presentation Evidence ### Intended Audience - **Technical AI practitioners / engineers**: Assumes familiarity with concepts like context rot, RAG, quantization, evaluations, and benchmarks — e.g., "if you work with long horizon tasks, you probably run into this issue of context blow[up]." - **Community/conference setting**: "This is a big room, so if you're in the back, don't hesitate to come closer" — indicates an in-person talk; the speaker notes AI Engineering as "home community for me before being the hyperloop." - **Cost-conscious builders**: References Coinbase's reported AI spend reduction and frames findings in token/budget terms: "bad memory is expensive because it spends more tokens." ### Demonstrated Scope of Content - **Hands-on local/on-device experimentation**: Detailed description of a personal rig — "an M3 Ultra with 96 gigabytes and 28-core CPUs, I'm using two models... a Qwen 27B quantised at 4 and the DeepSeek V4 flash," with evaluations running on a Mac in Tokyo controlled from her phone. - **Empirical benchmark reporting**: Cites specific experiments — "I ran over 68 questions," testing a policy ladder (no recall → vector RAG → decisions ledger → ranked recall → oracle), with a result that "the ranked-only ledger performed the best." - **Memory-system design framework**: Presents a "write–manage–read loop" mental model and a harness architecture (core traces block, recall block, archival block). - **Cross-model and cross-benchmark validation**: "This actually works on several models, not only on the Qwen 27B, but also the DeepSeek V4 flash... I also tried it on the Spider V2 benchmark." - **Practical guidance for practitioners**: "Consider the recall policy as a first-class metric... what type of memories do you want to store? How do you rank them?" ### Presentation Style (Observed) - Structured talk: motivation → setup → framework → experiments → results → recommendations → employer plug/hiring note. - Evidence-forward: grounds claims in concrete numbers, benchmarks (Xbench, Spider V2), and ablation descriptions. - Personal/anecdotal touches: fans around the overheating Mac, running evals on a flight — conveys practitioner credibility. ## Distinguishing Claims from Demonstrations | Type | Example | |---|---| | **Creator claim** | "We believe in the importance of sovereign AI" (channel/employer stance) | | **Demonstrated pattern** | Rigorous ablation methodology held constant ("The model is fixed across all the different tasks") | | **Creator claim** | "It's not just that it gives you better recall, it actually costs less" (stated result of her experiments) | | **Demonstrated pattern** | Reproducibility orientation — open-source repository reference ("over 30 runnable cookbooks... from Diamond") | ## Caveats for Channel-Level Synthesis - Single-talk sample; conclusions about the broader channel's content identity should not be drawn from this video alone. - No private motivations inferred; the hiring and sovereign-AI statements are direct quotes, not interpretations. # Delivery Observations: Stefania Druga — "Memory Harnesses for Long-Running Research Agents" ## Overall Tone - **Conversational and informal**: heavy use of filler words ("uh," "um," "like," "right?") throughout, giving a casual talk feel. - **Friendly and welcoming**: opens with audience care — *"if you're in the back, don't hesitate to come closer"* — and closes with an invitation: *"come talk to me."* ## Recurring Vocabulary & Phrases - **"Right?" as a check-in**: repeated rhetorical device, e.g. *"like when the model starts contradicting itself"* intro aside — *"you can think of memory as a write manage read loop, right?"*, *"we are seeing the local models like crossing the line, right?"* - **"Like" as a frequent discourse marker**: *"like a very big claim,"* *"like what are the type of memories that you want to store?"* - **"So" as a frequent sentence opener/transitions**: *"So that makes this issue of dealing with context rot a priority,"* *"So I'm testing this by..."* — appears dozens of times. ## Teaching Habits & Structure - **Audience problem priming**: starts from a shared pain point — *"if you work with long horizon tasks, you probably run into this issue of context blow [rot]"* — before introducing her solution. - **Mental model framing**: explicitly offers an abstraction for the audience: *"this is the mental model I want you to have in mind."* - **Ladder / stepwise presentation of conditions**: describes the recall modes as a progression — *"baseline is like not to use memory at all... next is to use rag... then... a decisions ledger... and last but not least... an oracle."* - **Anticipating audience questions**: *"and you're probably going to ask like why is the oracle not hitting the max and I'm going to explain that too."* - **Concrete anecdote for credibility**: the fan-cooled Mac story — *"it started to get hot. So I had my husband put fans around it. Um we're running out of fans."* - **Takeaway heuristic stated explicitly**: *"maybe a good heuristic to have here is that bad memory is expensive."* - **Call to action for practitioners**: encourages the audience to *"consider the recall policy as a first class metric."* ## Communication Patterns - **Hedging and self-correction mid-sentence**: *"this this matters now more than ever,"* *"what is this an example of, right? like memory when we design a harness for memory this is the mental model."* - **Contrastive findings framing**: pairs negative and positive results — *"when your task fits in context, the harness doesn't add much. However, if I start to run tasks that are longer term horizon... having a good memory harness really starts to pay off."* - **Named-example anchoring**: cites specific artifacts and benchmarks (Nature retraction example, Xbench, Spider V2, GLM, Coinbase CEO tweet) to ground abstract claims. - **Casual naming of components**: *"I have a what I call an oracle, but basically this is the ground truth."* ## Uncertainty Notes - Some transcript artifacts (e.g., "Dug," "DeepSync V4 flash," "Xbench") appear to be transcription errors of names; quoted fillers may over- or under-represent actual verbal habits. - Single-talk sample; patterns like "So" openers and "right?" check-ins are well-evidenced by repetition, but no claims can be made about delivery outside this talk. # Memory Harnesses for Long-Running Research Agents — Position Record **Speaker:** Stefania Druga (transcript renders "Dug"), Research Scientist at Sakana AI, Tokyo **Source:** Video transcript `01a15ca7-fb64-7642-bf26-2f12125c4560` ## Motivation and Framing - **Problem: context rot** in long-horizon tasks — the model "starts contradicting itself," redoes work it forgot it had done, or drifts from the user's questions. - Druga cites **METR projections** (as she describes them) showing a trend toward longer-horizon tasks and **fewer model releases**, so solving context rot becomes a priority as those trends converge. ## Why Local Models - Cites a tweet (two days old, per her) from the **CEO of Coinbase** describing reduced AI spend with increased usage via local models, better routing, caching, keeping context clean, and usage visibility. - Local models "crossing the line": she names **GLM**, and notes **DeepSeek V4 Flash** runs on M3 Ultra, with RAM still a bottleneck. ## Experimental Setup - **Hardware:** M3 Ultra, 96GB RAM, 28-core CPU; machine in Tokyo running evaluations continuously (with fans added after it got hot). - **Models:** Qwen 27B (quantized to 4-bit) and DeepSeek V4 Flash; model held **fixed** across tasks while recall-block variables change. ## Harness Design: Write–Manage–Read Loop - Mental model: memory is a **control loop around the model**, not just a database store. - Architecture: - Small research agents with **zero durable memory** — all memory comes from the harness. - A **core** of traces always shown to the agent. - A **recall block** and an **archival block** (cross-session information). - **Ladder of recall modes tested:** 1. **Baseline** — no memory at all. 2. **Vector RAG** — similarity-based retrieval. 3. **Decisions ledger** — records decisions per turn, prioritizable. 4. **Oracle** — ground truth; supplies the correct memory each loop. ## Findings ### Task 1: Literature Review (retracted Nature paper) - Corpus includes a Nature paper claiming 742,000 promising materials, later retracted; the retraction is a "needle" hidden among headlines and citations. - **Result:** everything fit in context, so memory added **no capability** — same performance with and without memory, only added cost. - **Claim:** when the task fits in context, the harness doesn't add much. ### Task 2: Xbench (long-horizon memory benchmark) - Answer located at step ~124; question asked at step ~500, outside the context window. - **Result:** with **ranked recall** (rank-only decisions ledger), the model answered correctly more frequently; the ledger performed best, better than a gating policy ("do you need memory or not"). - Over 68 questions, multiple cells and seeds. - **Oracle doesn't hit maximum** because it provides correct memory but doesn't force its use — the model can ignore or misread it. - Ablations (arbitrary examples, wrong step, most recent step) still favored the **ranked policy**. - Results held on **both models** and on **Spider V2 benchmark**. - **Cost claim:** good recall policy saves tokens; heuristic: **"bad memory is expensive"** — it spends more tokens and can send the agent the wrong way. ## Recommendations to the Audience - Treat **recall policy as a first-class metric**: - What memories to store and how to rank them. - How to design the recall function. - What survives repeated runs and multiple sessions. - Notes the broader memory technique landscape: **30+ runnable cookbooks** in an open-source repository (she attributes it to "Diamond"), spanning short-term/long-term memory, cognitive techniques, evaluation-as-memory, file-system retrieval through trained memory models. ## Sovereignty Trade-off (stated with qualification) - Running locally gave full control over data, traces, compute, and evaluations — an example of **sovereignty** — "but it comes at a cost": models run serially (no batch querying for DeepSeek V4 Flash), so evaluations take a long time and were still running on her Tokyo machine. - Frames sovereign AI as important to Sakana AI in Japan; **Sakana AI is hiring**. ## Unresolved Tensions / Qualifications Preserved - Memory adds cost with no benefit when tasks fit in context (explicit negative result). - Oracle underperforms expectations — correct memory is not sufficient without model compliance. - Sovereignty vs. throughput trade-off acknowledged but not resolved. # Video Attribution Record — Memory Harnesses for Long-Running Research Agents ## Speakers & Roles | Speaker | Role | Notes | |---|---|---| | **Stefania Druga** (spoken as "Stefania Dug" in transcript) | **Creator / Presenter** | Self-identified: "research scientist at Sakana AI in Tokyo." Delivers the entire talk; sole speaker. | - No guests, narrators, or other identified speakers appear. - Audience/room is addressed (Q&A implied: "you're probably going to ask"), and applause closes the talk, but no other speaker is transcribed. ## Referenced Sources & Third-Party Claims - **Meter projections** — "recent projections from meter" on longer-horizon tasks and fewer model releases. - **Coinbase CEO tweet** — described as two days old; claim that Coinbase reduced AI spend while increasing AI usage via local models, routing, caching, and visibility practices. *(Third-party claim, unverified within the talk.)* - **Nature paper retraction** — paper claiming discovery of "742,000 promising materials," later retracted; used as a corpus task. - **Benchmarks**: **Xbench** (described as "an established benchmark for long horizon tasks memory") and **Spider V2**. - **Models referenced**: Qwen 27B (quantized to 4-bit), "DeepSeek V4 flash" (transcribed as "DC V4 flash" / "DeepSc V4 flash"), GLM, M3 Ultra hardware. - **Open-source repository** — "over 30 runnable cookbooks… from Diamond" on memory techniques. - **Sakana AI** — employer affiliation; attributed the "sovereign AI" position to the company. ## Promotional Segments - **Hiring pitch** (closing): "we are also hiring… if you want to come join us in Japan, come talk to me." Explicit recruitment on behalf of Sakana AI. - Company positioning ("sovereign capability… very important for us at Sakana AI in Japan") functions as organizational promotion. ## Provenance & Availability Facts - All results are **self-reported experiments** by the presenter on personal hardware (M3 Ultra, 96 GB RAM, 28-core CPU); no independent verification stated. - Presenter notes a local-model limitation: **serial inference only** (no batch querying for DeepSeek V4 flash). - Quantified claim stated: **68 questions** on Xbench with multiple seeds/cells — exact figures not itemized in transcript. - **Comment availability**: no comment data supplied; comments unavailable/not addressable in this record. - Transcript contains **likely speech-to-text artifacts** ("Stefania Dug," "Meter" [possibly METR], "Xbench" [possibly XBench], "Diamond," "heristic"), flagged as uncertainty rather than corrected. Hello, welcome. Uh, this is a big room, so you're if you're in the back, don't hesitate to come closer. Um, my name is Stefania Dug. I'm a research scientist at Sakana AI in Tokyo. Uh I used to be based here and AI engineering uh is home community for me before being the hyperloop. So it's very good to be back and today I'm going to talk to you about memory harnesses for longunning research agents on device. So if you work with long horizon tasks, you probably run into this issue of context blow, right? Like when the model starts contradicting itself or it has to redo the work because it forgot it did that task in the first place or it starts to drift from your questions because it forgot them. And this this matters now more than ever because from this recent projections from meter we see that the trend is to solve longer and longer uh horizon tasks and also that we're getting fewer and fewer model releases. So at some point later this year we're going to have this convergence right where we'll get many more long-term horizon tasks and fewer model releases. So that makes this issue of dealing with context rot a priority. And why did I wanted to to tackle this problem on local models and with a local harness? Uh maybe some of you have seen this tweet. It's only two days old. Uh the CEO of Coinbase actually shared how their company managed to reduce their AI spent while actually increasing uh the AI usage. And the way they did that was by transitioning to use many more local models but also having better practices like using better routing, better caching, keeping the context clean and then having better visibility for what people are using and for what uh what kind of task. So we are seeing the local models like crossing the line, right? Like GLM is on everyone's minds like especially with Fable going away. uh DeepSync V4 flash can now be run on uh M3 Ultra and there's still a bottleneck for RAM. It's tricky, but these local models are starting to be useful for agentic tasks and for tool use. So, I wanted to show you what has been my setup for the experiments I'm going to share with you today. Uh this this is my Mac. It's still running evaluations right now uh back in my desk in Tokyo and I'm controlling it from my phone. Um and after running EVAs non-stop for a couple of days, it started to get hot. So I had my husband put fans around it. Um we're running out of fans, but the the machine is still running and the valves are still giving results. Um, on this M3 Ultra with 96 gigabytes and 28 core CPUs, I'm using two models. I'm using a Quen 27B quantise at 4 and the DC V4 flash. And before I show you how I built the memory harness on this machine, I wanted to tell you what this little what is this an example of, right? like memory when we design a harness for memory this is the mental model I want you to have in mind um you can think of memory as a write manage read loop so it's not just the database store it's actually this control loop around the model more concretely how did I take that loop and customized it so this is my harness design like I started with research agents that are the small agents because they have zero durable memory and I wanted all the memory to come from the harness And then um in the middle I have a core which is always shown to to the agent um of traces. And then I have a recall block where I'm testing different modes and an archival block where I'm keep keeping track of information across different um sessions. And in that recall block I'm actually going through a ladder of modes that I'm testing. The baseline is like not to use memory at all. no recall at all. So I'm I'm testing for that. Uh next is to use rag vector vector rag um just to see whatever like the harness would pull in terms of similarity. Then is to use a decisions uh ledger where I actually keep track of what decisions are being made for every turn and then I can prioritize them. And last but not least and this piece is very important. I have a what I call an oracle, but basically this is the ground truth. So this is like telling the harness for every loop what the correct memory that needs to be retrieved is. And the model is fixed across all the different tasks. So the only things that I'm changing is like these different variables in the recall block. And I wanted to to give you an example of a first task that I tested. So I wanted to see if I give the agent a task of doing literature review and I'm including a lot of papers in the corpus where there was a big scientific claim like this is actually a nature paper where they said they discovered 742,000 promising materials like it was a very big claim which got retracted later but the retraction it's a much smaller like haystack back needle in that corpus than the headlines and the citations. So I wanted to see if if the system can retrieve the right answer for these type of questions. And what I found was because like for these tasks all the papers and all the information fit into the context the memory actually didn't add more capability. It was the same performance with memory and without memory and it only added more cost. So when your task fits in context, the harness doesn't add much. However, if I start to run tasks that are longer term horizon and the entire task and the relevant context doesn't uh fit, then having a good memory harness really starts to pay off. So this is another example of a task that I ran. This is actually from an established benchmark for a long horizon uh tasks memory. It's called Xbench. And this is an example of a question, right? So I'm asking a question and then like the right answer is in a like step 124. But the moment when I ask the question, I'm asking it like at step 500. So it's completely outside of the context window and the model needs to use the memory harness to retrieve the specific answer from the right step. So I'm testing this by uh changing the different policy ladder that I explained before with memory off uh by deploying recall different types of recall and by using the oracle as a reference. And what I found was that with the ranked recall, the model gets the right answer um more frequently than without. And here is a breakdown of the decomposition of performance on this Xbench tasks. So I ran over uh 68 questions and for each of these questions there were like multiple um cells and lots of different seeds. And what I found was that the rank only ledger performed the best and it performed better than like just gating the harness by saying do you need to use memory or do you not need to use memory and you're probably going to ask like why is the oracle not hitting like the max and I'm going to explain that too. So the oracle what it does it provides the right information the right memory to the model but it doesn't force it to use it. So the model can get the right memory but still retrieve the wrong information or choose to ignore it or be confused. So that's why the oracle in this case doesn't hit the max performance. And I've done lots of ablations on these tasks to see like what happens if I give arbitrary um examples. What happens if I give it the wrong step? What happens if I give it the most recent step? And I still found that the best performing condition was the one with the ranked policy for recall. And this actually works on several models, not only on the Quen 27B, but also on the DS4 flash. And it also works across different benchmarks. I also tried it on the Spider V2 benchmark. And it's not just that it gives you better recall, it actually costs less. So maybe a good heristic to have here is that bad memory is expensive because it spends more token and it can send agent the wrong way. But having like a good structural policy for recall can save you a lot of tokens and uh budget. So one thing that I want to encourage you from this experiment is to consider the recall policy as a first class metric and to start to think about how you might use it in your systems. Like what are the type of memories that you want to store? What how do you rank them? Like how do you design your recall function? And then um what are the type what survives when you run this over and over and over? and um multiple sessions, multiple runs and this is just a simple first kind of experiment. Um but the memory technique landscape is very rich. Um so there's over 30 runnable cookbooks that are shared in this open-source repository from um Diamond and memory is complex. We have short-term long-term different cognitive techniques. uh we can use start to use evaluation results as well. Um and right now there's actually a a pretty broad landscape of solutions, right? So going from simple file system retrieval to training memory models. Um there's there's a wide spectrum of solutions from less structural to completely structured. Um so I think there's a lot of research we're going to see in this space. uh it's important and it becomes more and more relevant and for me it's been super fun to to test this on local models um because I got to control everything. I got to control the data I was using the entire traces of compute and evaluations and um yeah I I see that as an example of sovereignty and it comes at a cost. Uh I didn't tell you that these local models I can only what uh run them in serial like they don't support batch querying for the DeepSc V4 flash. So that's why I am still running evaluations back on my computer in Tokyo or I I was doing it on the flight on my way here because it takes a long time. Um, but I still think it's very powerful and it's a very good test for what memory can do when you can control every single step of the pipeline. And this sovereign capability is part of a bigger ecosystem that is very important for us at Sakanai in Japan. Um, we believe in the importance of sovereign AI today more than ever. And we are also hiring. So, if you're interested and want to hear more about this and if you want to come join us in Japan, come talk to me. Uh, thank you very much. \[applause\]