Talk by Soheil Feizi (RELAI, University of Maryland) on continual learning for AI agents, focusing on how agents can learn from production failures without drifting or breaking existing capabilities. It presents a framework of verifiable continual learning built on four requirements: converting failures into replayable learning environments, preserving prior capabilities during updates, routing repairs to the appropriate layer of the agent stack, and keeping the learning loop efficient for continuous operation. Current approaches are examined in light of these principles, including prompt optimizers, memory consolidation, coding-agent harness repair, and trace-to-harness systems, along with the remaining gap toward a holistic, lifelong, verifiable learning loop with online regression control. # Channel Identity Observations: Soheil Feizi / RELAI ## Explicit Self-Description - **Creator role stated directly**: The speaker introduces himself as **"Soheil Feizi. I'm founder and CEO at Rely. I'm also an associate professor in the computer science department at University of Maryland."** - **Company promotion embedded**: The talk repeatedly directs viewers to the company's product site: - > "if you're interested in any of the tools that I'll be talking in this presentation, you can visit at our website rely.ai" - > "if you want to try our VCL and apply it to your agent, you can use it today at rely.ai" - **Claimed organizational mission**: "this is what we have been working on at Rely to create a verifiable continual learning engine for AI agents based on these four principles." ## Demonstrated Content Scope - **Topic area**: Technical, research-oriented talks on **AI agent continual learning**, covering model fine-tuning (SFT, DPO, GRPO, LoRA), harness/prompt engineering, memory methods (LETA, MemZero), and a proposed framework ("verifiable continual learning" with four principles: **replayability, holisticness, lifelongness, efficiency**). - **Format**: Single-speaker presentation/talk style ("Today I'm going to talk about...", "That's it for today and for this talk"), including live product demonstration with benchmark results (e.g., "score increases to 97% from 87%"). ## Intended Audience - **AI/ML engineers and agent developers**: Evidence includes technical framing ("weights of LLMs," "regression-aware optimization," "pull request that you can review") and practical integration instructions: "you can add VCL to your current agent in just two commands," with agent built "on top of any of available major agent frameworks." - **Teams deploying agents in production**: Discussion centers on production logs, session feedback, regression traps, and reviewer workflows ("a reviewable version update... explaining what changes in the agent"). ## Claims vs. Demonstrated Patterns - **Creator claims** (unverified): product effectiveness ("10% improvement on average just with one loop"), ease of adoption, benchmark quality. - **Demonstrated**: The channel (at least this video) presents **founder-led technical content that doubles as product marketing**, blending educational taxonomy with a live Rely.ai product pitch and demo. # Delivery Observations: Soheil Feizi — "Continual Learning for AI Agents" ## Overall Tone & Register - **Academic-instructorial hybrid**: The talk blends lecture-style structure with a product pitch. Feizi opens with a personal credential framing — *"My name is Soheil Feizi. I'm founder and CEO at Rely. I'm also an associate professor in the computer science department at University of Maryland"* — establishing both authority bases upfront. - **Measured, low-affect delivery**: Few intensifiers; enthusiasm is conveyed through evaluative words like *"the beauty of it is..."* rather than emphatic exclamations. ## Structure & Signposting Habits - **Explicit roadmap statements**: He announces the agenda before diving in: > "I'll be talking about these two challenges, current approaches in order to deal with them and also provide some perspective of how we think about these two problems." - **Numbered framing is a recurring device**: "two fundamental challenges," "three layers," "four important principles," "three key takeaways." Each major section uses enumeration. - **Transition phrases**: *"All right, So..."* and *"So, let's get it started"* / *"let me dig a little bit deeper"* appear repeatedly as section pivots (e.g., lines [61], [142], [461]). ## Recurring Phrases & Vocabulary - **"Durable"** appears in the title, subtitle framing, and body: *"durable improvements," "the smallest durable change."* This is a signature term. - **The word "testable" / "verifiable"** recurs as a core evaluative axis: *"it is still not testable," "everything becomes testable and verifiable."* - **A framing-question pattern**: he poses a question, then answers it: > "Where does the feedback come from?" / "Is it enough? The answer is no." This rhetorical Q&A occurs at least three times. - **Layered taxonomy vocabulary** repeated verbatim across sections: **model layer, harness layer, memory layer** — introduced early and reused consistently through the talk. ## Teaching Habits - **Concrete failure scenario as illustration**: In the "holisticness" section he gives a worked example — *"an agent that cites a stale policy and skips the required escalation"* — then enumerates five possible root causes. This example-then-enumeration move is characteristic. - **Definition-by-contrast**: He distinguishes approaches by tradeoffs (expensive vs. cheap, testable vs. unverified, e.g., *"It can happen... the update is cheapest and fastest... but usually it is unverified"*). - **Self-referential loop-backs**: *"that touches the holisticness principle that I described"* — he explicitly maps product features back to earlier-taught principles, reinforcing structure. - **Live-demo narration**: During the demo portion he narrates results conversationally: *"Okay, the score is uh, not too uh, high. It is 78%."* ## Verbal Fillers & Disfluencies - Frequent **"uh"** and **"um"** throughout, sometimes clustered: *"in in this domain, in this layer"*; *"can uh can uh can be very can be very high."* This suggests a spoken, minimally polished delivery rather than scripted narration. - Occasional self-corrections: *"keep fast eligible refunds"* (likely intended "keep fast-track" or similar — uncertain given transcript only). ## Uncertainty Notes - Transcript-only evidence; pacing, volume, and visual emphasis cannot be assessed. - Filler-word density may be inflated or under-represented by automatic transcription; exact frequency claims should be treated as approximate. # Continual Learning for AI Agents: From Failures to Durable Improvements — Position Record **Speaker:** Soheil Feizi (self-identified: founder/CEO at RELAI (Rely); associate professor, Computer Science, University of Maryland) ## Core Thesis - The goal of continual learning is to let agents **learn from experience** — act, get feedback, improve — **without forgetting**, mimicking human experiential learning. - Two fundamental challenges frame the talk: 1. **Getting feedback** — how to know whether the agent did well, and what it should have done instead. 2. **Acting on feedback** — which agent layer/component to change and how. ## Where Feedback Comes From (Challenge 1) - **Development time:** benchmarks with evaluators produce grades (pass/fail/reward) and behavioral feedback. - **Production:** no benchmarks, only **session logs**, often without explicit user feedback. - Two feedback paths on logs: - **Automatic** — LLMs, models, or code analyze logs; even the agent can critique its own log. Scalable. - **Human expert** — lower volume, but provides domain expertise and alignment insight. - **Key claim:** log + feedback is *not enough* — it is not testable. What's needed is a **replayable learning environment**: a simulation with defined grading inferred from the single observation, involving decisions about real vs. mock tools, synthetic users, and inferred evaluators. ## Layers of Agent Improvement (Challenge 2) - **Model layer:** weight updates via SFT, RL post-training (DPO, GRPO, RLVR); LoRA makes updates cheaper and safer. Feizi notes these require benchmarks and explicit evaluators — not directly applicable to log+feedback alone — and are expensive. - **Harness layer:** prompts, skills, tools, code, workflow. - **Trace-to-harness:** a coding agent analyzes log+feedback to improve the agent; works on logs but is "wipe-based" (likely "hipe/pipe-based"), untestable, risks hidden regressions. - **GEPPO / prompt search:** mutate prompts, score candidates, keep winners via evolutionary search; testable but need benchmarks/evaluators. - **Memory layer:** store facts (LETA, MemZero), distill skills from successful trajectories. Cheapest and fastest; works directly on log+feedback, but usually **unverified** — no test that issues are resolved or regressions avoided. - **Guiding claim:** a good learning engine should seek the **smallest durable change at the right layer**. ## Verifiable Continual Learning (VCL) - Definition proposed by Feizi: improve an agent from its own experience where **every fix is proven to help and proven to break nothing that already worked**. - Three steps: **executable test** (failure becomes a replayable task), **measured delta** (update scored before/after), **regression test** (prior tests still pass). ### Four Principles 1. **Replayability** — turn one-off failures into rerunnable tests via learning environments. 2. **Holisticness** — one failure may have multiple causes (stale memory fact, unoptimized prompt, non-normalizing tool, missing escalation gate, weak model); route the fix to the layer explaining the failure. 3. **Lifelongness** — new fixes must not break past behavior; regression handled **within** the optimization, not post-hoc. Feizi adds an efficiency caveat: must not scale linearly with the number of past environments K. 4. **Efficiency** — the loop must run frequently; changes range from cheap (memory writes) to expensive (weight updates). ## RELAI's Implementation (Speaker's Product Claims) - Learning loop: signals (logs, feedback, instructions) → replayable learning environments → root-cause analysis routing fixes to memory/model/harness → regression-aware optimization → **reviewable version update** explaining changes. - Claimed integration: a one-time learning-harness setup plus **two commands** (create learning environment; optimize), producing an optimized pull request. - **Demo claims (fictional support-agent benchmark with deterministic evaluators and regression traps):** - From an instruction alone ("caller is rude and adversarial"), a learning environment with personas, intent, mock/real tools, and evaluators was generated; agent scored 78%. - After one optimization loop: **10% average improvement**, score rising from 87% to 97% (figures as stated). - Production-log example: feedback "keep fast eligible refunds, but do not generalize generosity beyond refund thresholds" lifted to a learning environment and optimized without regression; described as lifelong and compounding. ## Stated Takeaways 1. Agent continual learning is **not necessarily model fine-tuning** — many useful updates happen in harness and memory layers. 2. **Production logs are not learning environments** — they must be transformed into replayable ones. 3. The frontier is **regression-aware continual improvement** — fixing new failures while verifying old ones aren't broken. ## Notes on Source Character - The talk blends technical argument with product promotion (explicit CTA to rely.ai); RELAI benchmark results are self-reported and presented without external validation in the talk. # Speaker Attribution Record ## Video Metadata - **Title:** Continual Learning for AI Agents: From Failures to Durable Improvements — Soheil Feizi, RELAI - **Video ID:** `01a15ca8-d64e-76ce-8ed7-16a703938158` ## Speakers & Roles ### Soheil Feizi — **Creator / Sole Speaker** - Self-identified at start: *"My name is Soheil Feizi. I'm founder and CEO at Rely. I'm also an associate professor in the computer science department at University of Maryland."* - Delivers the entire talk; no other speakers, guests, or narrators appear in the transcript. ## Promotional Segments (Flagged) - **Opening promotion:** Directs viewers to company website: *"you can visit at our web at our website rely.ai."* - **Closing promotion:** *"if you want to try our VCL and apply it to your agent, you can use it today at rely.ai."* - **Mid-talk product promotion:** Extended demo of Rely's learning loop, including "you can add a VCL verifiable continual learning to your current agent in just two comments" and a benchmark demo showing claimed score improvements (78% → 97%, described as "10% improvement on average"). ## Referenced Sources & Third-Party Claims All methods named are cited by the speaker as existing approaches (no independent verification possible from transcript): - **Model-layer methods:** SFT (supervised fine-tuning), RL post-training (DPO, GRPO, RLVR), LoRA (low-rank adaptation) - **Harness-layer methods:** "trace-to-harness" approaches, GEPPO, prompt search with evolutionary algorithms - **Memory-layer methods:** LETA, MemZero, skill distillation ## Performance / Results Claims - Speaker reports benchmark results on a "fictional support agent" test bed; these are **self-reported claims from the company's own benchmark**, including a "regression trap" design detail. ## Provenance Limits & Uncertainty - **No curated metadata** was supplied beyond title; no speaker identities beyond the self-introduction. - **Comments availability:** Not stated in source material. - **Cited sources:** No external papers, datasets, or publications are formally cited; method names are mentioned without references. Hi everyone. My name is Soheil Feizi. I'm founder and CEO at Rely. I'm also an associate professor in the computer science department at University of Maryland. Today I'm going to talk about continual learning for AI agents. How we can go from failures to durable improvements. And if you're interested in any of the tools that I'll be talking in this presentation, you can visit at our web at our website rely.ai. Let's get this started. Humans learn mainly from experience by interacting with the world and getting feedback. The goal of continual learning is to imitate the same for agents. So they can also learn from experience by acting, getting feedback, and improving without forgetting. All right. So here's basically a bigger picture of how continual learning for agents look like. So here's an agent that interact with the world, with diverse users, with complex tools, uh with various data policies. And as I mentioned, the goal is to continuously improve the agent from its experience without forgetting. And this learning can happen in different layers of the agent. It can happen in the model layer where potentially we can change weights of LLMs or other models used in the agent, or use different types of models in the agent. It can happen in the harness layer where it brings the context, proper context to the LLM uh with components like prompts, skills, tools, code, workflow. And it can also happen in the memory, either in session memory or persistent memory of the agent. But there are two, I would say, fundamental challenges in continual learning for agents. The first challenge is how to get feedback. How do we know if the agent did well? And if not, what should it have done instead? That's basically the first part, getting the feedback. And the second part is how we can act upon that feedback. How we can optimize and improve agent and learn from that feedback. Which layer, which component do we need to change and also how. I'll be talking about these two challenges, current approaches in order to deal with them and also provide some perspective of how we think about these two problems. So, let's get it started with the first problem. Where does the feedback come from? So, the easy case is when we have a benchmark and some evaluators on that benchmark. So, the agent can run tasks from the benchmark. Now, we have the evaluators in order to score and we can get grades like pass, fail, or reward, as well as potentially some feedback on the agent behavior and agent performance. This is usually what is happening during the development time that different teams they curate benchmarks in order to understand the performance of the agent in certain applications. But in production, we don't have such benchmark. We have logs. Here's an example of a session log where a user is interacting with the agent. Maybe the user is not very happy with the way way the agent is behaving, but we don't have any explicit feedback. So, there are two ways of getting such feedback on such session logs. One is automatic using some other models or LLMs or code in order to analyze the log and provide feedback. In some cases, even the agent itself can look at it is log and provide some critics of it. It is automatic and it is scalable. And the second approach is where we have human experts to look at some handful of these logs and provide some domain expert expert feedback on those agent outputs. Uh this is lower in the volume, but it is critical because it provides expert knowledge on the behavior of the agent and its is alignment with the way that we want agent to behave in those applications. Either way, now we have session log plus some feedback on those logs. Is it enough? The answer is no because it is still not testable. Here we have log and feedback, but what we really need is a replayable learning environment, a simulation that we can rerun with defined grading on what success looks like, not one instance of what happened and the feedback on top of it. So what is a learning environment? Here we are inferring a distribution from one observation that replaces what happened and what success means. Uh the input is what we have some session logs and feedback. This is basically one observation of what happened and now we want to create a simulation and evaluation environment from that information that involves, for example, understanding how tools in the agent behavior in the agent log should behave, should we use real tools, should we use mock tools, and if so, what kind of data brought to the mocking process of those those tools. If the agent is interacting with some users, how we can infer synthetic users from that data. And also how success looks like in that learning environment. What are the evaluators that needs to be inferred. So there are lots of technical challenges in any of these components, but if we could do this successfully, they could use this the output as executable. We can now run different candidates of the agents against such learning environment, understand the behavior and the performance of the agent in those scenarios and in those patterns, and we can fix the issues um based on the feedback, based on the information that we observe because now everything becomes testable and verifiable. All right. So, the second problem is now we have this feedback, how we can act upon it, how we can optimize the agent. And from a high-level point of view, there are three layers that we can improve the agent. Uh there's a model layer where we can change the weights of the model, and there are methods like SFT, so supervised fine-tuning, uh RL-based post-training in order to make those changes. And these are usually expensive because that requires uh more intensive compute in terms of changing the model weights. The second layer is the harness layer, harness engineering, context engineering, where we can potentially pre-write prompts, maybe learn some skills, uh change tools or add tools, maybe change code around around the LLM, and there are different methods like get past race to harness, uh which provides a lot of flexibility in terms of learning from that feedback. And the last layer is the memory layer, where we store facts and learn skills uh in order to not repeat those issues uh and failures in in the future. But a good learning is not going to be focusing on any of these components exclusively. A good learning engine should ask for the smallest durable change at the right layer of the agent. All right. So, let me uh dig a little bit deeper into each of these layers. So, first in terms of updating the model weights, there are various approaches uh in order to do that, uh including SFT, supervised fine-tuning, where we imitate correct trajectories. Uh we often need labeled samples in order to um fit those samples, uh fit the model to those samples. Other approaches are based on RL, um reinforcement learning post-training, like DPO, GRPO, RLV R, where we sample score against the reward or preference signals, and reinforce what wins. And there are some categories based on LoRA, low-rank adaptation, that limits the set of parameters that can potentially change. It makes this uh the learning in this layer cheaper, and also safer um in terms of the updates. But these methods, they usually need benchmarks and explicit evaluators. They cannot be directly applied on, let's say, if you have a log and feedback, unless we turn those into uh replayable learning environments. Right, in terms of uh updating the harness, uh I would highlight the two uh categories in in this domain, in this layer. One is trace-to-harness approaches. Let's say you observe a log, you have some feedback on top of it, you can effectively ask a coding agent in order to analyze the log and improve the agent. So, this works on uh the case where we have log and feedback, but it is wipe-based. We don't know if even for that particular uh sample, if the change is effective, because it is not testable. And we don't know what is the impact of it on other samples and other scenarios. What uh might have been working previously, but with these changes might not work properly, and create some hidden regressions. The other category that I want to highlight is methods like GEPPO and prompt search where they mutate prompts, they score different candidates, and they keep the winners using some search algorithms like evolutionary algorithms. And these are These methods are testable, but they need a benchmark and explicit evaluators in order to have those have those scorings. And in the memory layer, we effectively write down facts and we distill skills so the agent doesn't rediscover them. It can happen in the information memory layer, methods like LETA and MemZero where they can effectively store a fact or correction. And we have also methods to skill distillation that compresses successful trajectory into reusable how-to do skill for the for the agent. So, this layer in terms of the update is cheapest and fastest. It works directly on the cases where you only have log and feedback, but usually it is unverified because you don't have a way in order to test whether or not writing in the memory will resolve the issues that you have dealt with and whether or not it can potentially create some regressions on some other cases. With that, let me introduce a new subcategory of continual learning called verifiable continual learning. In a verifiable continual learning, the goal is to improve an agent from its his own experience where every fix is proven to help and proven to break nothing that already worked. And usually it involve it involves three steps. Where we need to have an executable test where the failure becomes a task you can replay and create. Then we need to have a measured delta where the update is scored on the test before and after. And then we have a regression test. So prior test still pass even after we make such changes to the agent. So let's think about what are the principles of a practical verifiable continual learning first. And I will argue there are four important principles that we need to keep in mind. So the first principle is replayability. We need to turn a one of failure into a test that we can rerun. Here, as I had mentioned previously, many cases we have log and feedback, but that is not testable. We need to lift it in a learning environment to simulate and evaluate the agent on a similar pattern on a similar scenario. So everything becomes testable based on that simulation and evaluation environment. So that's basically the first principle that we need to have. The second principle is holisticness. One failure may have several causes and several possible repairs. You know, let me give you an example. Let's say you have an agent that cites a stale policy and skips the required escalation. The issue might come from the memory where you have some stale fact. It can come from uh not optimized prompt. It might come from a tool that doesn't normalize the policy. Might come from the workflow that we need to add escalation gate before we run. It might come from the model. Maybe the model is not a good model in order to have a strong reasoning. So here we need to route the fix to the right layers that explains the failure with the smallest durable change to the agent. And that is basically the principle of holisticness in verifiable continual learning. The third principle is lifelongness. A new fix must improve the new case without breaking the past. Let's consider the setup where we already optimize the agent on K past learning environments and a new failure comes and we turn that into a learning environment EK plus one. What do want to change? So, the first approach is okay, just focus on this new learning environment, but that can create regression on the past behavior on the past learning environments that the agent was successful. A better approach is a regression aware learning where the regression is not be treated as a post-hoc approach, but as a mechanism within the optimization itself. So, here we are uh fixing the recent failures subject to having no regression on the past uh learning environments. And obviously, this needs to be done in a efficient manner, so it doesn't scale even linearly with K because K can grow and the complexity of this approach can uh can uh can be very can be very high. And the last but not least principle is the efficiency. This continual learning loop needs to run frequently and we need to have efficiency in uh different layers in updates to the agent. So, sometimes the change can be cheap, like for example, writing something in the memory can be medium in terms of the complexity by changing the prompt or hardness, and sometimes it can be very expensive by changing the weights of the model. Also, efficiency should be in the optimization loop itself, especially when we have regression aware optimization and regression is treated within the loop not as a post hoc approach. To sum up, these are the four principles of a practical verifiable continual learning. Replayability, holisticness, lifelongness, and efficiency. And this is what we have been working on at Rely to create a verifiable continual learning engine for AI agents based on these four principles. In particular, here's how Rely's learning loop runs. You can start with some signals to this loop. It can be logs, feedback, or even instructions and prompts. We lift those in those signals to replayable learning environments. So, that's based on the replayability principle. This makes everything that follows testable and verifiable. Then we do root cause analysis and route the fixes to the right layer of the agent. It can be memory, it can be model, or it can be harness. That touches the holisticness principle that I described. We have regression aware optimization. Regression is not being treated as a post hoc approach, so that touches the lifelongness principle that I mentioned. And obviously, this loop should run efficiently. That touches the efficiency principle. So, the output of this is a reviewable version update to the agent explaining what changes in the agent during this loop and why those changes are are improving the agent without creating regression. The beauty of it is you can add a VCL verifiable continual learning to your current agent in just two comments. So, the first one is a one-time setup. You create a learning harness in your agent. You can use your own LLM and your agent can be built on top of any of available major agent frameworks. And then after that, you need two commands in order to activate this learning loop. You can create learning environments using various type of signals that you can have either log log feedback or some instructions, and then you can call to like optimize in order to use holistic lifelong optimizer to improve the agent. And the output is a optimized version pull request that you can review and you can use it in order to improve your agent. So, let's look at the how it actually works in practice. We build a continual learning benchmark on a fictional support agent case where we have reproducible test beds for continual learning in a tool using support agent. So, we have a single source of truth and the policies are interacting for this agent to be handling. So, we have deterministic evaluators and we also build this benchmark in a way that it has some regression trap. So, if the optimizer focuses on overfitting on the latest fix, it can potentially break what are what the agent was previously successful on other tasks. So, let's say we have an agent. We don't even have logs or anything, and we want to just see how the agent is behaving. Uh, let's say when um, caller is rude and adversarial. So, simply we can create a learning environment using such instruction. And what it will do, it will create a learning environment to simulate and evaluate the agent. Uh, the simulator will include personas, intent, mock, uh, or real tools and also the learning environment contain evaluators in order to define success metrics. All of these are produced from just one interactive comment. And after that, we can just simulate the agent using this learning environment, see how it behaves. Okay, the score is uh, not too uh, high. It is 78% and in particular, there are two um, evaluators that uh, basically show very low scores uh, of agent in this environment. So, these are some of the failures that we observe. How to improve such failures? We can do that by calling Rely Optimize with certain number of rollouts. And as you can see, the average improvement uh, can be um, quite high. Uh, it is 10% improvement on average just with one loop and score increases to 97% from 87%. Okay, now let's consider the case that now the agent is in production. Now, you have a log, you have an agent session that isn't um, not desired and you have a feedback. For example, you can say keep fast eligible refunds, but do not generalize generosity beyond refund thresholds. So, that's a feedback on the agent behavior. Again, the flow is the same, so we lift it into a replayable learning environment and we call Rely Optimize in order to mitigate this uh, issue. Use this feedback without creating regression of the agent behavior in past environments. And this is lifelong, so you can keep doing that uh to improve the agent without breaking what already works, and it is compounding. This is verifiable continual learning in practice, where each update is tested, every gain is measured, and nothing that already works breaks during this optimization. So, that's it for uh today and for this talk. So, there are three key takeaways that I want to highlight. The first one is agent continual learning is not necessarily model fine-tuning. The updates and many useful updates can happen in the harness and memory layer. So, the second takeaway is production logs are not learning environments. We need to transform them into replayable learning environments to simulate and evaluate the agent on the same patterns and scenarios. And the third takeaway is that the frontier is regression-aware continual improvement, where when fixing the new failure, we verify that we don't forget the old ones. We don't create regression. So, that's verifiable continual learning built on four principles: replayability, holisticness, lifelongness, and efficiency. And if you want to try um our VCL and apply it to your agent, you can use it today at rely.ai. Thank you.