Talk by Benedikt Sanftl (CEO and Co-Founder of Mutagent) introducing the "agentic AI engineer": a multi-agent system, coordinated by an orchestrator, that builds AI agents inside an eval-driven development loop. The approach combines spec, build, evaluate, diagnose, monitor, and optimize stages, analogous to coding agent loops but applied to building agents. The talk concludes with a live demo of one of the company's agents in research preview. # Observable Identity Evidence — "The Agentic AI Engineer" (Mutagent) ## Explicit Self-Description - **Company affiliation claimed directly**: Benedikt opens with *"I'm Bene. CEO and co-founder of Mutagent"*; Burak introduces himself as *"the CTO of Mutagent."* - **Product claim**: *"as we at Mutagen work on this, we're going to show you now how this looks like as a product"* — the talk positions Mutagent as a product company, not just a content channel. - **Deployment model stated**: *"for now, everything runs in your environment. And we offer it as cloud and local, um, we'll offer a managed service down the line."* - **Product scope stated**: *"a set of agents. We have two in research preview"* — specifically an **evaluator agent** and a **diagnostics agent**, connected by an orchestrator *"which runs in your coding environment."* ## Observable Presentation - **Format**: A conference-style talk with two speakers alternating, including a live/pre-recorded product demo section (*"let's show our first agent in research preview to the audience"*), closing with *"have a great conference."* - **Intended audience**: AI/ML engineers and teams building and operating agents in production. Evidence: - Assumed familiarity with agent concepts: *"as you all are aware of now, loops is the hot topic"* - References to observability tooling (LangFuse), coding harnesses (Claude Code/Codex, *"I'm using code code"*), GitHub PRs, ticketing systems, Slack. - Frames problem in org terms: *"in your organization you're not planning to roll out hundreds of agents."* - **Claimed topical scope**: The full agent lifecycle — spec-driven development, build, eval-driven development, ship, monitoring, diagnosis, and optimization — unified as the **"Agentic AI engineer"** concept. - **Positioning as experts**: Burak cites experience: *"building agents for the last 3 years."* User-derived claims: *"we learned from our users that reading through these traces took them a lot of time."* - **Promotional intent observable**: The talk blends educational framework (offline/online loops, eval design, diagnostics) with product demonstration and a call to action: *"reach out to us if you have further questions."* ## Notes on Distinction - Statements about what the product **does** (HTML artifact generation, multi-tier trace filtering, orchestrator) are creator claims demonstrated via described demo, not independently verified. - No channel-wide mission statement beyond this talk's scope was present; identity evidence here is limited to a single conference presentation. # Delivery Observations — The Agentic AI Engineer (Benedikt Sanftl & Burak, Mutagent) ## Format & Structure - **Two-speaker tag-team format** with clear role division: Bene handles the framing/introduction, transitions, and product close; Burak delivers the technical deep dives and live demo. - Bene repeatedly uses **short steering questions to hand off**, e.g.: - > "Okay, let's continue. What happens after build?" - > "Why would I want to change the platform in a year down the line? So, what what what did we learn out of experience here?" - > "why is it so good to use this kind of concept or thought process in the building?" - Burak answers with **"Right."** or **"Yeah, so…"** as a consistent turn-opener: > "Right. So basically then the spec tells your coding agent what to build…", > "Right. So, in the context of agents in general, the mainly the trajectory is important…" ## Tone - **Conversational, informal delivery** from both speakers, marked by filler words ("um", "uh", "you know", "let's say"): - > "Um yeah, you implement the change. Um yeah, you generate some samples…" - > "This can be a code update. This can be a direct update on a any agent platform or again your local harness agents." - Non-verbal artifacts appear in the transcript ("[snorts]" at [84] and [415]), consistent with a casual live-talk register. ## Teaching Habits & Recurring Devices - **Analogy to familiar software practices** is Burak's core explanatory device, used repeatedly: - Evals as tests: > "can think of essentially the equivalent to unit tests for coding." - > "the eval-driven development loop, which I would call. And this is kind of equivalent to test-driven development…" - **"Again" as a discourse habit** — Burak inserts it constantly to re-anchor points: - > "Uh again, this can be based on the volume of traces…" - > "And here again, this is only the definition stage." - > "The reason being, apart from coding agents, the agents in other domains they handle specific processes…" - **Enumerated stage-by-stage walkthrough**: the pipeline (spec → build → eval → ship → monitor → diagnose → optimize) is introduced, then each stage expanded in sequence — a recurring structural loop in the talk. - **Concrete-number illustration** to justify a claim: - > "imagine you have a data set item of 200. Here, without automated evals… takes quite a while." - > "if you have let's say millions of agent traces and to read all of these uh it actually costs more than the execution itself." - Bene uses **"as we learned (now/before)"** to recapitulate: - > "as we learned now the details of each everything starts with a spec." - > "you diagnose them as we learned before with the diagnose agent…" ## Vocabulary & Framing - **Central recurring metaphor: "loops"** — offline loop, online loop, optimization loop, agentic loop, "you design loops for your agent." - **Recurring framing of the human as bottleneck**: - > "the bottleneck basically becomes the human review and the human yeah, building time." - > "Then the human essentially becomes the bottleneck." - Domain jargon used freely and repeatedly: *spec, harness, traces, evals, failure modes, root cause analysis, LLM as a judge, orchestrator, vibe implement*. - **Named-tool references** ground the talk in the current ecosystem: "cloud code or a codex agent", "Hermes coming up, deep agents", "LangFuse", "GitHub", "Whisper flow". ## Interactive Patterns - Bene occasionally **redirects mid-flow for clarity**, e.g. > "So, before we continue, Burak, um we have two passes here… should we dive deeper for like a half a minute…" - Burak's answers often end with a **circular callback**: > "And then the whole cycle repeats again." - The demo segment shifts to **second-person walkthrough style** ("Here I have some start commands… if I type diagnose here, this will basically start…"). ## Closing Delivery - Bene closes with **imperative, slogan-like phrasing**: - > "stop uh, the debugging. Um have agents do the tedious work and reach out to us if you have further questions." ## Uncertainty Notes - The filler density and "again" habit are consistent across both speakers' long turns, so they appear **stable patterns**, not isolated moments. - Snorts and disfluencies appear only twice; whether they are habitual beyond this talk cannot be established from a single transcript. - Audience reaction is not captured in the transcript, so pacing/engagement effects cannot be assessed. # Position Extraction: "The Agentic AI Engineer" — Benedikt Sanftl & Burak, Mutagent ## Speakers - **Benedikt "Bene" Sanftl** — CEO and co-founder of Mutagent - **Burak** — CTO of Mutagent ## Central Thesis - **Bene**: Loops are the current dominant paradigm for building software; applying the same loop concept to *building AI agents themselves* is the natural next step. The **"Agentic AI Engineer"** — a set of agents automating the agent development lifecycle — is proposed as the solution. - **Burak**: Once an organization reaches a certain number of agents, the human performing the loop cannot scale in time; doing the loop agentically increases throughput by fitting more cycles into the same time window. ## Two Loops - **Offline loop** (Bene): while building — iterate, test, evaluate, improve the agent. - **Online loop** (Bene): once deployed — monitor traces, diagnose, feed back into the optimization loop to iterate across agent versions. ## The Problem: Manual Loop Bottleneck - **Bene**: Current manual lifecycle: issue → implement change (possibly "vibe implement" with coding agents) → generate test samples → review results/traces → ship → A/B test. Feedback is manual and slow. - **Bene**: The bottleneck is **human review and human building time**; this doesn't scale, especially when rolling out hundreds of agents. ## Lifecycle Stages (Burak) 1. **Spec**: define responsibilities, functions, decisions, conditions — definition only. 2. **Build**: realize the spec in a harness/framework or as a Claude Code / Codex agent. 3. **Evaluate**: define evals — "the equivalent to unit tests for coding." 4. **Ship**: deploy as code update, platform update, or local harness update. 5. **Monitor**: continuous monitoring with trigger conditions (trace volume, weekly/daily jobs). 6. **Diagnose**: collect failures, structured root cause analysis. 7. **Optimize**: create specific changes/"mutations" for found failure modes; cycle repeats. ### Two Paths - **Burak**: **Cold start** (build from scratch, starts at spec) vs. **existing feature** (agent already running, optimize over it). ## Spec-Driven Development (Burak) - Goal: capture requirements and especially **success criteria**; agent behavior is company- and environment-specific. - Spec must define: context requirements, integrations/tools, jobs to be done (and not done), constraints and boundaries. - Spec becomes a **blueprint** that implementation is held against; it is deliberately **isolated from implementation details**. ### Why platform-independence matters (Q&A) - **Burak** (from 3 years of building agents): frameworks/harnesses sometimes lack capabilities; you hit bottlenecks and depend on the framework to fix them, which "can take a while." Flexibility to pick the best harness is key. - **Bene** corroborates: new harnesses shipped in recent months; the field moved from agents building code toward "agent loop runtimes," citing **Hermes** and **deep agents**. ## Eval-Driven Development (Burak) - Equivalent to test-driven development; evals are the **termination condition** — "when is an agent good enough?" - Two ways to build an eval suite: - Sit with domain experts to write metrics/criteria upfront — but Burak acknowledges teams "will already know that this is a bit difficult": **you cannot pre-guess the entire evaluation suite**. - Start with historical or synthesized data. - **Burak's key claim**: the real, complete eval suite is a **product of discovery** — collected over time from user feedback and production failures, including edge cases. ### Why an evaluator agent? - **Burak**: with a 200-item dataset, manual evaluation via observability dashboards takes long; with many features it becomes impossible; **the human becomes the bottleneck**. - **Bene/Burak**: design loops with clear eval/termination gates so agents run autonomously in the background; the engineer's job becomes designing these loops. ### What good evals look like (Burak) - Agent evaluation focuses on the **trajectory**: was context complete; chain-check every tool output (wrong tool output → wrong final output); the harness itself affects agent behavior and is an optimization vector — evaluate all of these, not in isolation. - Evals must provide **actionable feedback**: score-based LLM-as-judge evals don't tell you what to fix unless the rubric is very well defined; **binary/criteria evals** are preferred because failure gives a call to action. - **LLM-as-judge must be calibrated** against scoring noise; since LLMs are non-deterministic, variance must be handled or you cannot conclusively say an improved version is better. ## Online Loop: Diagnostics (Burak) - Failures recur; identify failure modes, group by root cause (prompt section, missing/malfunctioning tools), then generate new evals to detect problems and remedies to fix them. - **Learned failure modes accumulate** as historical data each agent can check against. - Upfront cost: deep-reading traces initially; over time you collect **code-checkable indicators** (specific content/tool-call sequences) to diagnose without reading all traces. - Reading millions of traces costs more than execution itself; aim for **intelligent segmentation** and representative sampling. ## Autonomous Optimization (Burak) - With an eval suite, vary the feature, run "auto-research style" experiments; if target scores are reached, **automatically shipped to production**; production issues feed remedies back; deploy while eval suite is green. ## Full Lifecycle Summary (Bene) - Spec → build (agentic) → offline loop (evaluate/optimize/test) → deploy → online loop (user feedback, monitoring, diagnosis agent doing root cause analysis and failure clustering, deriving new eval criteria that become part of the spec). Agents "continuously grow with you"; more production data → better-scoring agent. Concept transfers from software-driven development with coding agents. ## Product Reveal (Bene & Burak) - **Bene**: everything runs in the user's environment (cloud and local); managed service planned. Two agents in research preview: - **Evaluator agent** — builds eval sets/datasets, core of the eval-driven loop. - **Diagnostics agent** — diagnoses production traces; motivated by user feedback that reading traces took too long; can be spun off from the coding environment. - Connected via an **orchestrator** running in the coding environment; connectors to trace sources (e.g., LangFuse, local transcripts, exported JSONL, ticketing systems, Slack failure reports); outputs include GitHub PRs, MD file adjustments, deployments. ### Demo details (Burak) - Orchestrator shows a dashboard of stages, agents, prior configurations; start commands trigger workflows (e.g., "diagnose"), scoped to an agent or a skill. - Diagnostics retrieve traces from configured sources; **Burak acknowledged the run takes "quite a while"** and showed a pre-generated result instead. - Output: generated **HTML artifact** with agent details, tools, harness; multi-tier filtering/segmentation picks representative trace samples; LLM reads a portion to detect obvious problems, then focuses on a failure mode. Alternatively, user-guided search for a specific reported issue. - Report contains: detected issues with frequency, per-failure-mode explanations, a **"recursive why chain"** tracing origin, remedies, and an **assumptions block** — Burak: important because the diagnostics agent may make incorrect assumptions when it lacks code access, which users can correct. - Final output: a **markdown task definition** for the coding agent to apply fixes; feedback box supports speech-to-text (Whisper Flow mentioned). ## Closing (Bene) - Vision statement: "stop the debugging" — have agents do the tedious work; invites contact. # Speaker Attribution & Provenance Record **Video:** The Agentic AI Engineer – Benedikt Sanftl, Mutagent **Video ID:** `01a15ca8-da45-7d60-9351-95d8f0dc5a04` ## Speakers & Roles | Speaker | Role | Affiliation | Evidence | |---|---|---|---| | **Benedikt "Bene" Sanftl** | Co-presenter / **Creator-side speaker** | CEO and co-founder of **Mutagent** | Self-introduction: "I'm Bene. CEO and co-founder of Mutagent" | | **Burak** (surname not stated) | Co-presenter / **Creator-side speaker** | CTO of **Mutagent** | Self-introduction: "I'm Burak. I'm the CTO of Mutagent" | - No **narrator**, **guest**, or audience speakers appear; format is a two-person conference talk. - Speakers alternate without timestamps in the transcript; speaker turns are marked by `>>` but not consistently attributed by name. ## Promotional Segments (Flagged) - **Product pitch — Mutagent platform:** Benedikt explicitly transitions to product: *"as we at Mutagen work on this, we're going to show you now how this looks like as a product."* - **Research preview agents:** Two agents promoted as in "research preview" — the **evaluator agent** and **diagnostics agent** — connected through an **orchestrator** running in the user's coding environment. - **Demo segment (Burak):** Walkthrough of the diagnostics agent producing an HTML artifact, multi-tier trace filtering, "recursive why chain," remedies, and markdown task definitions for coding agents. - **Deployment/availability claims (promotional):** - "everything runs in your environment" - Offered "as cloud and local" - "we'll offer a managed service down the line" *(future claim, not currently available)* ## Referenced Sources & Third-Party Claims - **Third-party tools/platforms named in passing** (no claims evaluated): - Coding agents/harnesses: **Claude Code** (transcribed as "code code"), **Codex** - Agent frameworks: **Hermes**, **deep agents** (cited as examples of new harnesses) - Observability: **LangFuse**, generic JSONL trace exports - GitHub (automated PRs) - **Slack** (as a source of failure reports) - **Whisper Flow** (speech-to-text, cited as example feedback box use case) - **No formal sources, citations, or external evidence** are cited; all content is the speakers' own claims and product descriptions. ## Third-Party Claims About Users - Benedikt states: *"we learned from our users that reading through these traces took them a lot of time"* — an unverified user-feedback claim by the company. ## Provenance & Availability Notes - **No curated metadata was supplied** with this source; all provenance derives from the transcript alone. - **Comment availability:** Not stated in the transcript; unknown. - **Timestamps:** Not present in transcript; speaker turns cannot be precisely located. - **Claims of personal experience:** Burak cites "building agents for the last 3 years" as basis for platform-flexibility advice. - **Future/uncertain commitments:** Managed service is explicitly framed as future ("down the line"), not a current offering. Hi everybody. Welcome to our talk the Agentic AI engineer. I'm Bene. CEO and co-founder of Mutagent and I'm here with my colleague. \>\> Hi, I'm Burak. I'm the CTO of Mutagent and today we're basically going to talk about loops and how the Agentic AI engineer works. \>\> So, as you all are aware of now, loops is the hot topic how you build software in an Agentic loop. And we apply the same loop to the building of AI agents. And as you all aware, there is uh two concepts here. One is the offline loop where while you build, you iterate um on your agent, you test it, you evaluate it, you improve it, and you go on. And then you have a second loop which is we call the online loop where once the agent is deployed to production, you monitor its traces, you diagnose it, and then you feed it back into your optimization loop uh yeah, to iterate and have multiple versions of your agents. Yeah, up to until now um what we did was uh just doing this loop manually. It's quite slow. Um the life cycle is basically um you have an issue, um you want to change something to your agent. Um yeah, you implement the change. Um \[snorts\] you maybe vibe implement the change if you use coding agents for it. Um yeah, you generate some samples for this new feature or issue to test it. Um yeah, then you look at the result, you look through the traces, how does the outcome look like. Then you maybe ship it, you do AB testing, and all your feedback is kind of manually, it takes very long. Um yeah, and uh the the bottleneck basically becomes the human review and the human yeah, building time. And yeah, that you can't scale especially if in your organization you're not planning to roll out hundreds of agents etc. Yeah, and uh yeah, this is why we think the Agentic AI engineer is the natural next step to build agents. And I'll have Burak deep dive into how we improve timing and the road to production reliability with the Agentic AI engineer. \>\> Uh so, yeah, the key thing here is basically once you reach a certain number of agents or AI-based features, the human performing this loop again cannot really scale in enough time. So, this is why doing this agentically is the key to increasing the throughput because then you can fit many more cycles into the same time window. And now how that loop works is basically we have a few stages. So, this is when you are starting from scratch like the current software development practices, you first create a spec for your agent or your skill in this case. And here you need to define all the responsibilities and the functions that agents needs to handle, the decisions that it has to make on certain conditions. And here again, this is only the definition stage. Once you defined your agent's requirements, then you can finally go on to the build. And build is where you then realize that spec in a specific harness or uh agent framework or in these days you could even build it as a cloud code or a codex agent. Then comes the next step. This is where you define clear evaluations to evaluate your agent's performance because these are the key metrics then where you can say, "Hey, my agent is functional or not." Uh can think of essentially the equivalent to unit tests for coding. This is how you verify your agent works. Then after evaluation, if everything looks fine, you usually have the ship basically where you deploy this agent to production. Uh again, this can be a code update. This can be a direct update on a any agent platform or again your local harness agents. Then comes the online part. This is where then the agent is continuously monitored for issues and based on certain trigger conditions, then you can start automatic diagnostics. Again, this can be based on the volume of traces that your agent generates or, you know, weekly or daily jobs. Uh then we go into diagnosis stage. This is where you collect all the failures for your agents and do structured root cause analysis to then understand where the failures are coming from. Once you understand and categorize the failures, then you can finally go on to the optimization stage. This is where then you create, let's say, very specific changes or mutations for your agents and to deal with the found failure modes. And then the whole cycle repeats again. You evaluate and if everything looks good, then you can deploy again. Now, we will maybe do a deep dive on each stage, what that entails. Then \>\> So, before we continue, Burak, um we have two passes here. Like one is the cold start path, and one is basically existing features. Um should we dive deeper for like a half a minute on what differences here, what we see, and why this is important. Yeah. \>\> Yeah, so today, if you again sit down to build an agent, there are, you know, two options. Either you already have an agent, it's built, it's already running somewhere with um, you know, certain accuracy. However, the other option is again you are creating from scratch. So, when you create from scratch, obviously you design with the spec and the conceptualization stage. If you have an existing feature, the most likely again the agent is there, but then you are optimizing over something that already exists. \>\> Okay, cool. And \[snorts\] Yeah, let's dive deeper into each phase. I mean, you just mentioned the specking of new building agents, so this seems to be an important artifact, so let's dive into it. \>\> Right. So, with the spec-driven development, also prevalent for building software artifacts these days, or any kind of coding agent workflows, basically the goal is to capture the requirements for the agent, and especially the success criteria. The reason being, apart from coding agents, the agents in other domains they handle specific processes and this differs from company to company, so it's very specialized to the environment where the agent is in. And then the key point here is to clearly define which context requirements that the agent has, again which integrations and tools it then needs to have. What are the jobs to be done or the responsibilities that the agent will handle and what it will not. And then finally again the constraints and in general the boundaries for the agent. \>\> Okay. \>\> Now once we can define a clear spec as I said it becomes like a blueprint for any future development which then the implementation is held against. \>\> Okay. And let's dive into how we would then build an agent from the spec, right? \>\> Right. So basically then the spec tells your coding agent what to build and then here the target platform choice is entirely yours. Cuz as I mentioned agent space is very changing very rapidly these days. So the framework you are using today, you might want to change, you know, in a year or so. So essentially because of that spec is isolated from the implementation details. So once you decide to you can pick any target. Here then your coding agent of your choice will take that spec and give you a, you know, initial version of that agent, which is then basically customized to run on any platform that you see fit. \>\> Why would I want to change the platform in a year down the line? So, what what what did we learn out of experience here? Um \>\> Uh again building agents for the last 3 years, sometimes the agent framework or the harness does not always have the capabilities. So, occasionally you hit um like a bottleneck or a roadblock, and then you have to rely on the underlying framework to kind of get rid of that, and this can sometimes take a while. The key here is to be flexible because yeah, essentially again, it you want to pick the best harness or the framework that can, you know, fulfill your requirements. \>\> Mhm. I mean, we've all seen it in the in the last months, we how new harnesses shipped and yeah, how everything went from like a agents building code towards yeah, defined as like a agent loop run time. I mean, we've seen Hermes coming up, deep agents, and all of these frameworks, right? \>\> Yeah. \>\> Okay, let's continue. What happens after build? \>\> Now, after you build for agents, you essentially go into the eval-driven development loop, which I would call. And this is kind of equivalent to test-driven development for building software with agents because then the agents needs uh termination condition, right? So, when is an AI feature or an agent is good enough? So, here again there are two ways to kind of create your eval suite, which is composed of the evals like the metrics and the criteria you evaluate. And the data sets themselves, which usually contain the cases you have to satisfy. Now, in the beginning, the original option is that you can sit with your domain experts and then try to write down eval metrics and criteria that would cover the feature or the agent you want to build, but most teams working on that will already know that this is a bit difficult as in you cannot pre-guess the entire evaluation suite from the beginning. And secondly, you can always start with historical data or synthesized data from a known you know, sample of the data that you would like agent to be tested on. But essentially, the real and the complete eval suite is a product of discovery. What that means is over time, from user feedback, from production failures, you collect the metrics and criteria plus the additional data set cases, uh which is often representative for edge cases or you know, hard cases that the agent needs to deal with. And with that, then you finally have an evaluation suit where you can run the agent against an exactly know where it fails. \>\> Okay, before we continue, why does the Agentic AI engineer help us here so much? So, um obviously we're talking about the evaluator agent. So, why is it so why is it so good to use this kind of concept or thought process in the building? \>\> Well, I mean one issue is again, imagine you have a data set item of 200. Here, without automated evals, running this and evaluating by like human eyes takes quite a while. Um you would have to again scroll through an observability dashboard and logs and this in turn increases your loop time per eval state. So, as soon as you have a lot of features that you need to evaluate and experiment on, then it suddenly becomes impossible to, you know, do this quickly or in parallel. Then the human essentially becomes the bottleneck. \>\> Okay. So, we just have an agent do this work in sifting through traces. \>\> Yes, as the current era says a bit, you know, um you design loops for your agent so then they can autonomously work as many of these things in the background. Uh and then your job becomes designing these loops with a clear eval or termination gate. \>\> Okay. Understood. So, let's look at the how good eval uh supposed to be constructed and what it kind of evaluates. \>\> Right. So, in the context of agents in general, the mainly the trajectory is important because agents receive input or let's say intent or tasks and then they have a specific system prompt or like a decision tree that they kind of operate over. And when doing agent evaluations, again in general, we check, "Hey, was the context complete?" As in, did the agent have all the required context to perform the task end to end? Then this includes also chain checking every, you know, tool output in the trajectory because every wrong tool output in session can in the end lead to our wrong output as the final output. And apart from that, again, there are different things you can evaluate on. So, these days also the harness that the agent is operating on has quite drastic effects on the agent behavior. And this is again another vector of optimization. But in the end, when you evaluate an agent, you would like to evaluate all of these things and not something in just isolation. \>\> Mhm. \>\> Right. And in general, here, what makes an eval useful? So, you can always have you know, metrics or evals that are working in a LLM as a judge fashion and give you some score, but here, in order to make an eval, you know, useful, it has to provide actionable feedback. Uh, as in when you use score-based evals, unless your rubric is very well defined, then this does not exactly tell you what to fix. Uh, in such cases, using uh, binary type of evals or criteria, is preferred because there you have a kind of a call to action. If an eval or a criteria fails, then you know exactly what happened and how to kind of deal with that problem. Then, uh, another point is your LLM as a judge solution should be calibrated so that you don't have the, uh, scoring noise between judges because since LLM LLMs are undeterministic, what you will mostly encounter is the same judge can, uh, evaluate a problem different ways on each run. And then here, you have to make sure your LLM as a judge solution deals with this variance problem. Otherwise, it's hard to run experiments and conclusively say, "Hey, my initial or my improved version is better than my initial version of my agent." \>\> Mhm. \>\> Right. Uh, after then evaluation, basically, we go live and this is where then, uh, collecting, let's say, learned failures, failure or primary signals from the execution takes place. Um here the idea is again when an agent encounters a problem over time, there will most likely be multiple occurrences of these. So, it starts by identifying what failure mode uh that the agent has encountered and then grouping essentially these failure modes by the root causes and where they originate from. Again, this could be a section in the agent prompt. This could be missing tools or, let's say, malfunctioning tools. But, after your diagnostics results are categorized, then you can finally generate new evaluations. First, to detect these problems. And then, second, you can generate improvements and remedies based on these problems. And yeah, with that, over time, you have um build-up of these learned failure modes. So, then every agent over time uh gathers these historical data that it can always check against. When you uh start diagnosing, there's a bit of a upfront cost in the beginning because you often need to deep read the LLM traces to find out what's going on. Over time, you can collect code-checkable indicators per failure mode. What that means is there are maybe specific pieces of content or there's a specific tool called sequence where you know that the agent will encounter an issue and then this later on helps you to diagnose problems in your traces without actually having to read through all your traces. Now, why is that a problem? Because if you have let's say millions of agent traces and to read all of these uh it actually costs more than the execution itself. So, it's not the most efficient way of diagnosing the problems. Uh what you want to aim for is again try to uh pick a representative sample from your um whole traces by you know intelligent segmentation strategies and using again as I said the learned indicators. \>\> Okay, cool. \>\> And then with that you can finally start building the autonomous optimization loop because here then given that you have a eval suit, you can you know vary your feature again update certain or even run auto research style experiments to see that whether you can reach the desired or the target scores for your evals. As long as you can reach that then it's automatically shipped to production. Then from production you have your outer loop. Again, every issue that's found then on diagnostics gives you a improvement or a remedy which then you can optimize on and then as long as the eval suite is green then this again gets deployed to production. \>\> Let's continue now with the whole life cycle. Um yeah, as we learned now the details of each everything starts with a spec. Um you define and design it. Uh you build your agent and all of this can be done obviously agentic. So you define agents that do this work. And then they all together become uh the agentic AI engineer. Um you go through the offline loop where once you build your evaluation system uh yeah, you evaluate, you optimize, you test, you improve your accuracy on your test data set. Once you feel ready, you deploy it to production. And that's when the online loop starts. Here you get real feedback from users. Um yeah, uh you review the test cases, you continuously monitor them, you look at your live traces, you diagnose them as we learned before with the diagnose agent you can do root cause analysis, you can cluster your failure modes. Uh you'll derive new eval criteria from it. They become part of the spec. Uh they become part of the agent. So they they continuously grow with you while you use your agent in production. And uh yeah, this becomes the online loop. And the more use cases you collect, the more is production data you see, the better your agent becomes and the better scoring your agent becomes. And this all together now um becomes one end-to-end loop that you can run agentic. And uh yeah, from here on uh yeah, you can now see the combination to software-driven development with coding agents. You can transport this concepts also for AI engineering and building AI agents. And uh now, as we at Mutagen work on this, we're going to show you now how this looks like as a product. Um This is kind of how it looks like. Um for now, everything runs in your environment. And we it as cloud and local, um we'll offer a managed service down the line. Um so, it's a set of agents. We have two in research preview. The ones we talked deep about this in our talk. First one is the evaluator agent that helps you build an eval set, a good data set, because this is the core of the optimization, the eval driven development loop. The second one we have is the diagnose the diagnostics agent. It does you It helps you diagnose your traces you already have in productions, because we learned from our users that reading through these traces took them a lot of time, and having kind of they can just spin off from their coding environment to analyze the traces is very helpful to them. And as you can see here, um both of these agents um are connected through an orchestrator, which runs in your coding environment. And what you need basically is connectors to sources where you basically have all your traces, where you get your incidents from. This could also be like your ticketing system, um Slack where people report failures of your agents. Then you connect them. And then, obviously, as we mentioned before, there's different target platforms. Um this could be come a PR that automatically gets raised in GitHub. Um this could be just the adjustment of your agents in uh in MD files. Uh yeah, different frameworks we target. And or being deployed to managed services. Um yeah, kind of this is how our platform works. We spin up different agents. They are connected to an orchestra and I guess Burak, let's show uh our first agent in research preview to the audience. \>\> Uh in practice, as I said, you have um orchestrator which kind of handles the dispatch for all the other sub agents or stages we have. And then you can use it in any coding harness or agent you want. In this case, I'm using code code. But then what it will do is when you first boot up, it will show you kind of like a dashboard of which stages you have and also kind of the existing things in uh let's say your code base, your agents, and the prior configurations. Now here I have some start commands. These are basically, let's say, trigger commands for the workflows or specific stages. And if I type diagnose here, this will basically start the diagnostic stage for the root cause analysis. And here again, you can point the diagnostics into a specific scope. This can be an agent. This can be a skill as well. So you could basically diagnose all invocations of a particular skill. Here uh when you start the diagnostics, it will uh retrieve traces from your configured source platform. In this case, again, it can be something like LangFuse. It can be your local cloud transcripts or another you know, like a JSON L format that's like exported from one of the observability platforms. Now, the diagnostics run itself takes quite a while to finish, so I will show you instead a pre-generated kind of version of what it does and how that looks like in the end. Okay, so what do we see here now? So, basically, once you run the diagnostics agent, as I said, on your agents, you will get a um, generated HTML artifact, which basically shows you, you know, the details of your agent, which tools you have. Uh, then in general, if you if you have code access, it will also tell you which harness or the framework that the agent is using. And then here, in general, it will basically go through your traces and then select primary signals, or let's say failure modes, depending on how frequent they occur in the trace samples. Now, as I said, most of the time you don't want to read all of your traces because this is not cost-efficient. So, there the diagnostics use like a a multi-tier filtering or segmentation to kind of pick a representative sample, so uh, originally, a few, you know, pieces of the traces, or let's say, portion will be read by the LLM to see that if there any detectable obvious problems. And based on that, then diagnostics can decide or to focus on a particular failure mode or a signal. Uh the other option is then trigger the diagnostics, you can also specify an issue that you have seen yourself or issue that you're looking for. This could be something that the users reported. Uh in this case, it's again more of a guided search, meaning hey, if you tell the diagnostics agent you have a particular problem you're interested in, then it will try to find all incidents or occurrences regarding the same problem. \>\> Okay. And what do we see here in uh in yeah, in full here? \>\> Uh so, this is in general the overview, which tells you again which issues kind of were detected and kind of shows you like the frequency giving the given time frame. Then for each tab, you will have a failure mode and kind of like an explanation of the problem. And then here, you can kind of see where that problem comes from or why it happens. The agent will provide kind of like um uh recursive why chain, let's say, to then tell you, "Hey, this is where the issue is originating from." Now, in addition, it will offer you kind of remedies or fixes to deal with that. And then here, you will see an assumptions block. Uh here, this is important because uh when you are reading traces, sometimes you don't always have access to the code, so this helps us detect uh, LLM or let's say the diagnostics agent makes certain assumptions that are also not correct and we can see and correct them here. Uh, but in the end you will be presented by certain corrections or remedies, which then kind of can fix your problem. You can, you know, pick either the recommended or, you know, as many as you need as it's multi-choice. And once you basically go through all your problems, uh, at the end you kind of get to a decisions page and here again for those using, you know, text-to-speech or speech-to-text AI features like Whisper flow, there's always a general feedback box so that you can always talk into your mic. And finally, when you are happy with all the decisions, you get a markdown, let's say, um, task definition for your coding agent. So, all the fixes or the remedies can then be directly applied to once you go back to your terminal and to your coding agent. \>\> Thanks for the demo, Burak. So, last words on our product. Yeah, so, uh, thanks for listening into our talk. Um, yeah, I hope we could show you some, uh, good insights on how we envision the future of agent building with the Agent AI engineer. Um yeah, stop uh, the debugging. Um have agents do the tedious work and reach out to us if you have further questions. And yeah, have a great conference.