Ankur Duggal of Arize AI demonstrates using a coding agent with Arize's CLI and skills to evaluate and improve a flawed financial agent. The talk shows how traces reveal the supervisor, specialist agents, model calls and tools behind an answer, and how the agent investigates grounding problems, fabricated responses, redundant searches and silent failures. It covers building dataset, evaluator and experiment skills to test proposed fixes, including an earlier fix for international stock queries that required updating the underlying tool, not just the country context. Topics include local experiment execution, judge models, evaluation costs and sampling, and the risk of focusing on linting over the real defect. The live repair is still in progress at the close. Duggal recommends human review of proposed changes and application-specific evaluations to detect later regressions. The description is drawn from the creator's summary; the outcome of the live repair is not shown. # Channel Presentation Observations: Single Video ## Source Scope - **Video:** "Let Your Agent Cook: Using Skills to Evaluate and Improve Your App": Ankur Duggal, Arize AI - **Available sources:** Title and transcript only. No curated metadata was supplied. - **Limit:** These observations describe one video and do not support channel-level conclusions. - **Transcript caveat:** The transcript appears automatically generated. It has apparent errors, such as the self-introduction "My name is Encore" against the title's speaker name, and "Cloud codeex" and "Alex or sorry Cloud Code." These are treated as transcription artifacts, not evidence of presentation choices. --- ## 1. Explicit Self-Description Statements the speaker makes directly about the talk or themselves. - **Framing of the talk:** > "I'm going to be teaching you how to make your agent cook. Again, not food related." - **Stated purpose:** The talk is about moving from "vibes" to evaluation-based shipping. The speaker describes this as a central problem: "you can't really ship based on this feeling of, 'Yeah, it's correct.'" - **Self-positioning:** The speaker presents the work as a team effort: "the evaluations me and my team had set up prior to this." No further biographical claims are made in the transcript. - **Self-disclosed caveat about demo practice:** > "I am using dangerously skip permissions. Please don't do this. This is not actually good practice. This is just for a demo." --- ## 2. Intended Audience (Demonstrated) The material indicates an audience of developers and practitioners who build or ship LLM agents. - **Audience evidence:** - The speaker polls the room on whether attendees "are building agents or have even shipped agents." - Concepts are explained for practitioners: traces, APM, LLM-as-a-judge, codebase evals, guardrails, and experiment loops. - The speaker invites technical questions throughout and answers them in depth. - **Format evidence:** The transcript opens with "Good afternoon everyone," and includes audience Q&A, which indicates a live session or conference-style presentation. --- ## 3. Claimed Scope What the talk covers, based on the speaker's own framing. - **Evaluation workflow:** Identifying production failures, building evals, running experiments, and shipping fixes in a repeated loop. - **Tracing as foundation:** > "that trace kind of helps you define that." - **Coding-agent automation:** Connecting a coding agent (Claude Code is used in the demo) through an Arize CLI and skills to pull traces, evaluations, and annotations, then propose and test fixes. - **Generalization claim:** > "Realistically, you don't have to use cloud codeex. You can use whatever sort of agent you want." --- ## 4. Demonstrated Presentation Patterns What the material itself shows, separate from the speaker's claims. ### Live demonstration - The speaker runs a live financial-advice agent scenario and walks through a real Claude Code session. - Outputs are narrated as they arrive, including "We'll let it cook." - Some results are shown from a pre-recorded or earlier run, which the speaker discloses: > "I'm going to go ahead and show you a previous run because again this will take some time." ### Problem-first framing - The talk starts with a concrete failure case: a wrong tool call in a refund scenario. - The speaker then asks the audience to diagnose a failure in a financial agent before showing the fix. ### Quantified claims made during the demo - About 44% of interactions resulting in a bad summary. - A third iteration reaching about a 90% success rate, with the speaker noting "Clearly still not 100%." These figures come from a single demo run in the speaker's environment and are presented as illustrative, not as benchmarks. ### Hedged or balanced statements - Benchmarks: "Benchmarks are great... But realistically, what you want is, is it a good LLM for me?" - Limits of agents: "they're not magic. Like he said, they can definitely focus on linting." - Honest admission of uncertainty: "I genuinely don't even know what the right answer is to this." ### Vendor and product presence - The presentation is delivered by an Arize representative, and the platform is shown directly. - Product references include Arize, Phoenix (described as "Arize's open-source version"), and an enterprise edition with "online evals" and "dashboarding." - The speaker states product limits as well as strengths, noting that some features are enterprise-only. --- ## 5. Claims vs. Demonstration | Claim | Evidence in this video | |---|---| | Coding agents can automate the full eval-and-fix loop | Shown in a live demo; results are narrated, not independently verified | | Skills keep agents focused and avoid linting-only fixes | Speaker states "genuinely I haven't seen that so much when using the skills"; no comparative evidence shown | | Evals detect regressions over time | Stated conceptually ("two months later because there's a model regression"); not demonstrated | | Improvement from 0% to ~90% pass rate | Shown in a single demo run with a custom dataset | --- ## 6. Limits of This Observation - **Single source:** Only one talk is available, so no patterns across the channel can be inferred. - **No metadata:** No view counts, descriptions, playlists, or channel self-description were provided. - **No private motives:** The speaker's reasons for choosing this talk or product emphasis are not stated and are not inferred. # Delivery Observations: "Let Your Agent Cook" (Ankur Duggal, Arize AI) **Source basis:** Title and transcript only. No curated metadata was provided. The transcript is auto-generated and contains recognition errors (for example, "Encore" as the speaker's name and "LM" for LLM). Quotations below reproduce the transcript as written. --- ## 1. Tone and Register ### Conversational and informal The speaker addresses the room directly and uses casual phrasing throughout. - Opening: "Good afternoon everyone. My name is Encore. Uh I'm going to be teaching you how to make your agent cook. Again, not food related. I don't know." - Reassurance on a risky step: "I am using dangerously skip permissions. Please don't do this. This is not actually good practice." ### Light self-deprecating humor Small jokes and admissions of imperfection appear at several points. - "I apologize. Oh, here we go. I know what I'm doing, guys." - "Oh, did not mean to scroll there." ### Emphatic hedging on uncertain points When the speaker is unsure, the transcript shows explicit hedging rather than confident claims. - "I genuinely don't even know what the right answer is to this because this agent is still messed up right now." - "I'm being pretty vague here but essentially I'm just telling it go ahead and make the changes that you recommended" --- ## 2. Recurring Vocabulary and Phrases ### Intensifiers and softeners - **"realistically"** recurs as a framing word: "But realistically, you can't really ship based on this feeling of..." and "Realistically, you don't have to use cloud codeex." - **"genuinely"** appears in several sentences: "Genuinely, I haven't seen that so much when using the skills." - **"pretty"** is used as a generic intensifier: "It's pretty long, pretty manual, and pretty tedious." ### Simplicity framing Setup steps are repeatedly described as easy, even when the speaker concedes the topic is complex. - "I know this may sound really complicated... it is very simple to get it set up." - "Super easy. It would just be a couple steps to get that set up." ### Tag-style confirmation The speaker often closes a point with a short confirmation prompt. > "Right? There's so many different areas that we need to test and go against." > "You're not making those fixes just based on, hey, I know this is wrong and the output's incorrect." ### Stock acknowledgments Audience questions are typically met with a fixed opener. > "Great question. Yeah. So, you would go ahead..." > "Very very good question. Um so eval can be set up however you like." The phrase "great/good question" appears well over ten times in the Q&A segments. ### Verbal fillers "Um," "uh," "yeah," and "okay" appear frequently, especially at transitions and while the demo runs. --- ## 3. Teaching Habits ### Interactive opening The talk begins with a show of hands to gauge audience practice before introducing content. > "I'm just curious how many of you currently are shipping based off vibes." The speaker then uses the responses to frame the problem: "That was a solid hand up, which is kind of what I expected." ### Audience-sourced problem diagnosis Before revealing the solution, the speaker solicits guesses from the room and restates them. > "I want a little bit of kind of uh voices from this side of the room... what do you guys think could be the actual issue." Responses are summarized in the speaker's own words: "So what she said was, 'Hey, maybe the agent needs to ask a lot more questions...'" ### Analogy to familiar territory New concepts are anchored to traditional software engineering. > "If you've ever used traditional software engineering, you have something that you do call traces. It's for APM monitoring." The contrast is then drawn explicitly: traditional code is "very easy to follow," while in an agent "all you have is that one line of code, which is just going to be agent.run and then the query." ### Live demo with narrated wait time The speaker fills processing delays with explanation and reframes them as part of the lesson. > "We'll let it cook. You know, it's kind of the point of this." > "I don't want to have you guys staring at a screen where Claude's just discombobulating or whatever it does in its free time." To reduce dead time, the speaker shows a previously completed run as a substitute. ### Explicit signposting Transitions are announced in advance. - "So before we get into eval, what I want to kind of take is a whole step back..." - "So what I'm going to do here is we're going to go ahead and look at a specific issue." - "So as a kind of a decent example as to what a trace would look like would be this." ### Periodic recap Key points are restated near the end. > "So as a kind of refresher as to everything we kind of covered today..." > "That's kind of the general gist of what I wanted to show today." ### Check-ins for understanding The speaker repeatedly checks whether the audience is following. - "How is everyone doing? How do you guys feel? Do you guys have any questions about how you can go ahead and get this started?" - "Did that help?" ### Concrete product-level reassurance When asked about the tool itself, the speaker tends to answer with capabilities rather than abstractions: "It can basically do anything that you that you would be able to do in the platform." --- ## 4. Communication Patterns Around Caveats and Claims ### Repeated framing of tools as not magic The closing section restates a limit more than once. > "These AI agents, they're great. They will do basically everything you ask them to, but again, they're not magic." Related point: "The emails are only there to help you determine where there are issues, monitor those issues, make sure they don't repeat." ### Assertion stated with personal certainty Some factual-seeming claims are prefaced by the speaker's own assurance. > "I promise you it told me that." > "That's a very good understanding and that's kind of why we built out these skills..." These are assertions about the demo itself and are not independently verifiable from the transcript. --- ## 5. Stable-Looking Patterns vs. Isolated Moments **Appears repeatedly (more stable-looking):** - "Great question" / "good question" as a response to audience questions - "realistically" and "genuinely" as framing words - Narration during agent processing delays - Recapping and signposting transitions - Use of "right?" as a confirmation tag **Appears once or sparsely (isolated):** - The dangerous-permissions warning - The Phoenix open-source comparison - The discussion of benchmarks and LLM selection - The "Encore" name error (transcription artifact, not a delivery choice) --- ## 6. Limits of This Evidence - This is a single talk. Patterns are observed within it and cannot be generalized to the speaker's other presentations. - The transcript is auto-generated, so some word choices may reflect transcription errors rather than what was spoken. - Audio cues such as pacing, emphasis, and tone of voice are not available. Observations about "humor" or "hedging" rely on text alone and reflect how the words read, not how they were delivered. - Several Q&A responses are only partially captured because audience questions are often inaudible in the transcript, so the speaker's answers sometimes lack their full context. # Observation: "Let Your Agent Cook: Using Skills to Evaluate and Improve Your App" **Source:** Video `01a15ca6-a422-7585-ab4b-13b1e185d63a` **Presenter:** Ankur Duggal, Arize AI (per title metadata; the transcript opens with "My name is Encore," an apparent transcription error) **Other speakers:** Unnamed audience members, identified only as "a question" or by unnamed voice markers. Their suggestions are recorded as audience contributions, not as the presenter's positions. --- ## 1. Core Thesis (Presenter) The presenter argues that agent improvement should rest on **observability and evaluation**, not on informal checks. - **Vibe-based shipping is a starting point, not a sufficient basis for production.** The presenter says it "is not a bad thing" and "gets you going," but "you can't really ship based on this feeling of, 'Yeah, it's correct.'" - **Evals are hard to build** because agent behavior is "not a deterministic use case" and "you don't really know what you're looking for." - **Traces are the foundation for evals.** Traces let you see each step (LLM calls, retrievals, tool calls) that a non-deterministic agent takes. > "You won't know what your agent's doing. So that trace kind of helps you define that." --- ## 2. The Improvement Loop (Presenter) The presenter describes a standard eight-step process, which they characterize as: > "pretty long, pretty manual, and pretty tedious. It does it does work, but we want to kind of iterate and improve that." **Proposed alternative:** Consolidate data sources (traces, evals, human annotations, APM traces, codebase, and tickets) into a coding agent (Claude Code, Cursor, Codex). The presenter says this can move the process "into minutes." **Stated claim:** The presenter says the workflow is "very simple to get it set up," requiring "just a couple steps" for the CLI and skills. --- ## 3. Demonstration Case: Financial Advice Failure ### Observed failure - A simple financial agent was asked whether to invest in Tata Motors and answered "yes." - The presenter states they are "not registered to give any financial advice, nor is this agent," and raises legal concerns. ### Diagnosis proposals (from audience, not verified) | Suggestion | Attributed to | Presenter's response | |---|---|---| | Agent should ask more clarifying questions | Audience member | Accepted as "a very good guess" | | More context, prompt changes to gather context | Audience member | Acknowledged | | Lack of validation or ground truth | Audience member | "You're right." | | Agent doesn't know user's financial background | Audience member (question) | "In this case, no, it does not." | **Qualification:** The presenter explicitly says, "I genuinely don't even know what the right answer is to this because this agent is still messed up right now." They also state "there's genuinely not one correct answer." **Proposed fixes mentioned:** Ask more questions, add a guardrail so the agent responds only to certain questions and does not give unwarranted financial advice. --- ## 4. Claude's Analysis (Reported, Not Independently Verified) The presenter ran Claude Code against the trace data and reported findings. These are Claude's outputs as displayed, not validated by the presenter. ### Reported findings - **~44% of agent interactions produce a bad summary.** The presenter: "That's really bad. I did not know that." - **Fabricated answers**, described as "just as bad as" the summarization problem. The presenter says the two "may be related, it may not be." - **Redundant tool calls**, described as wasteful of time, resources, and money. The rate is described only as "relatively high." - **Silent tool failures:** the system reports success but gives no actual response. - **Hallucination when tools fail** instead of disclosing failure. - **Numeric groundedness issue**, which an audience member had previously flagged as a possible problem. ### Proposed actions (Claude-generated, endorsed by presenter for testing) 1. Build a tool-success silent-failure eval. 2. Fix numeric groundedness so responses are tied to reference points. 3. Test changes against the spans that caused the original failures. --- ## 5. Experiment Results (Cross-Market Stock Queries) ### Context - Users asked about stocks outside the U.S. (mostly South Korean, per the presenter, with the assumption that "there's other answers here" too). - The presenter notes that "people are asking that, but we don't support it." ### Iteration history (as described) 1. **Baseline:** Buggy, 0% pass rate, matching production failures. 2. **Iteration 2:** Normalized by adding country name. Did not work. 3. **Reason for failure (per presenter):** The tool being called was outdated; it needed updating to a stable version. 4. **Iteration 3:** Reported "about a 90% success rate." **Qualification:** The presenter says: "Clearly still not 100%. We still need to make some changes." ### Caveat on timing - The presenter said the process was "a couple minutes," then said the Claude run "took about 14 minutes." - A previous run was shown to avoid waiting on the live one. --- ## 6. Evaluator Design - **Where experiments run:** Locally in the presenter's demo, which they describe as "what will be the case most of the time." Cloud hosting is also possible. - **Where evals run:** On the Arize platform. - **Evaluator types:** One codebased and one LLM-as-judge. The presenter notes the codebased evaluator "kind of failed in getting the codebased data to set up," so the LLM-as-judge was the final version. Claude built it. - **Summary evaluator:** Reviews the full context given to the summary agent and checks whether the summary references the right points and answers the question. The presenter says this is how they judged a summary as "bad." ### Model choice (Presenter) - Teams can use any model, including open-source, local, or cheaper ones. - Some teams deliberately use top-tier models for evals even when production uses cheaper ones, because "they know their agent's only going to be as good as their evals." --- ## 7. Skills vs. Evals (Presenter's Position) Responding to an audience question about whether skills alone could ensure good summaries: > "The emails are only there to help you determine where there are issues, monitor those issues, make sure they don't repeat." - Skills and prompt changes are the fix. - Evals detect **regressions**, such as "a model regression because people start asking questions in a different way." --- ## 8. Coding Agent Behavior ### Linting concern (audience member) An audience member said coding agents often focus on linting rather than real issues. - **Presenter agrees:** "that's a very good understanding." - **Presenter's response:** The skills are designed to keep agents focused on the reported issues. Trace data shows that calling the wrong tool "is not going to happen because of linting." - **Qualified claim:** "Genuinely I haven't seen that so much when using the skills." This is the presenter's experience, not a controlled finding. ### Human oversight (Presenter) - The presenter says the human should check that the agent "didn't just go back and fix the linting changes" and that changes actually improve the agent. --- ## 9. Other Positions and Claims ### Phoenix availability (Presenter) - Phoenix is Arize's open-source version. - The presenter believes Phoenix also supports skills and MCP. - Enterprise adds "online evals" that are easier to set up, more dashboarding, and other features. ### Cost monitoring (Presenter) - Platform traces show the cost of running evals. - Evals can be sampled rather than run on 100% of traces or calls. ### Benchmarks (Presenter) - Benchmarks are "great" for general quality, but the presenter says "realistically no" to a benchmark for judging eval-model quality. - The presenter's stated principle: "if you're going to use out of the box eval to judge your agent you're going to get out of the box results." - Teams must conduct their own analysis to find the right model for their use case. ### Permissions disclosure (Presenter) - The presenter used a "dangerously skip permissions" flag and says "Please don't do this. This is not actually good practice." It was used only to avoid repeated approvals during the demo. --- ## 10. Presenter's Summary Takeaway > "These AI agents, they're great. They will do basically everything you ask them to, but again, they're not magic." The presenter describes coding agents as able to automate the full workflow (observing issues, building datasets, creating fixes, testing), with a human still responsible for verifying that changes help. --- ## 11. Unresolved Tensions - **Speed vs. verification:** The presenter promotes "minutes" instead of days, but the demo ran 14 minutes and produced an iteration still at ~90%, not 100%. - **Automation vs. oversight:** The presenter says agents can automate the workflow but also says humans must verify results. - **Claude's findings vs. validation:** The 44% bad-summary figure and other failure rates come from Claude's analysis. The presenter does not independently confirm them, and the financial agent's correct behavior is explicitly undefined ("I genuinely don't even know what the right answer is"). - **Linting focus:** The presenter agrees the concern is valid but reports not seeing it much in practice. - **Vendor framing:** The presenter works at Arize and demonstrates Arize products. Claims about ease of setup and capability come from that context and are not independently tested in the video. # Speaker Attribution and Provenance Record **Video ID:** `01a15ca6-a422-7585-ab4b-13b1e185d63a` **Title:** *Let Your Agent Cook: Using Skills to Evaluate and Improve Your App*, presented by Ankur Duggal, Arize AI **Source channels:** Transcript only. No curated metadata beyond the title was supplied. --- ## 1. Speaker Roles ### 1.1 Primary Speaker (Presenter) - **Stated name in title:** Ankur Duggal - **Stated affiliation in title:** Arize AI - **Transcript self-introduction:** "My name is Encore." This conflicts with the title. It is likely a transcription error, but the transcript does not confirm it. - **Role:** Presenter, product demonstrator, and speaker throughout. The presenter describes Arize products (CLI, skills, platform features) and runs the live demo. - **Speaking scope:** Opening remarks, concept explanations, live demo narration, and answers to audience questions. > **Identity flag:** The name in the transcript ("Encore") does not match the title ("Ankur Duggal"). Attribution to Ankur Duggal rests on the title alone. ### 1.2 Audience Participants - Several unnamed audience members answered polls and asked questions. - Some participants are referred to by pronoun only ("she," "he," "his question"). No names are given. - Audience contributions include hypotheses about the agent's failure (e.g., needing more clarifying questions, lacking validation or ground truth, not knowing the user's financial background). - **Attribution limit:** Individual identities cannot be established from the transcript. Some questions are inaudible or partially transcribed. ### 1.3 Attributed Non-Human Sources These are not speakers, but the presenter attributes findings to them: - **Claude / Claude Code:** Presented as the coding agent performing trace analysis, eval construction, and experiment runs. Its outputs (issue lists, percentages, eval definitions) are reported by the presenter. - **Agent under demonstration:** A financial agent (described as a supervisor agent plus a financial data agent). Its outputs are shown or described live. --- ## 2. Referenced Sources and Third-Party Claims ### 2.1 Tools and Platforms Named | Source | Context in Transcript | |---|---| | Arize (platform) | Trace storage, evals, datasets, experiments, annotations | | Phoenix | Described as Arize's open-source version with skills and MCP support | | Claude Code | Coding agent used in the demo | | Cursor, Codex | Mentioned as alternative coding agents | | Jira | Mentioned as a ticketing source that could feed the coding agent | | APM tools | Mentioned as system-health monitoring sources | ### 2.2 Specific Figures and Claims Attributed to Demo Outputs - **"About 44% of agent interactions result in a bad summary."** Presented as an output of Claude's analysis of platform evals. The presenter says "I did not know that." The transcript does not show the underlying data. - **Numeric groundedness issue, fabricated answers, redundant tool calls, silent tool failures, hallucination when tools fail.** Reported as Claude's findings from its analysis. - **About 90% success rate** after a third iteration on a South Korean stock dataset. Reported from a prior run. The presenter states it is "still not 100%." - **0% baseline passing rate** on the initial production-based traces. Reported from the prior run. - **Tata Motors and Nvidia queries.** Used as example prompts. The earlier financial-advice output is described verbally; the presenter says it is "not directly in the screenshot." ### 2.3 Third-Party or External Claims - Benchmarks: The presenter states that no benchmark dataset exists for choosing an evaluation LLM ("realistically no"). This is a claim about availability, not a sourced fact. - Model release cadence: The presenter says AI models "ship every other day." This is an unsourced general statement. - Linting tendency: An audience member's observation that coding agents tend to focus on linting is affirmed by the presenter, who says "I haven't seen that so much when using the skills." This is anecdotal. --- ## 3. Promotional Segments > **Promotion flagged.** The presenter is affiliated with Arize AI, and the talk promotes Arize products throughout. Promotional segments include: - **Arize platform walkthrough:** Traces, evals, datasets, experiments, annotations, and dashboards. - **CLI and skills setup:** Presented as "very simple" with a copyable install line and a profile setup. - **Marketplace skill installation:** Described for Claude Code. - **Phoenix:** Presented as the open-source version. The presenter says the enterprise edition has "nice to haves" such as online evals, easier formatting, and dashboarding. - **Model-provider integrations:** The platform is described as supporting a range of providers, including local open-source models. - **Direct product claims:** "It's very simple to get it set up." "It can basically do anything that you would be able to do in the platform." --- ## 4. Provenance Limits and Unavailable Channels ### 4.1 Evidence Not Present in Transcript - **Screenshots and slides:** Referenced repeatedly ("this slide," "in the screenshot") but not included in the transcript. Visual claims cannot be verified. - **Live demo outputs:** The Claude analysis, eval tables, and experiment charts are described verbally. Some were shown on screen but are not captured in text. - **Prior-run artifacts:** Previous runs are referenced but not reproduced. - **Underlying data:** The 44% figure, the 90% success rate, and the 0% baseline have no supporting data in the transcript. ### 4.2 Metadata Availability - **Comment availability:** No metadata was supplied. Whether comments are enabled cannot be determined. - **Speaker bio or credentials:** Not supplied beyond the title. - **Publication date and recording context:** Not supplied. ### 4.3 Audience Attribution - Audience questions are transcribed with limited identifiers. Several are partially audible. - Some questioner statements are only paraphrased in the presenter's response, so their exact wording is unknown. --- ## 5. Summary of Attribution Confidence - **High confidence:** The presenter's role and the demo structure. - **Moderate confidence:** The presenter's name, given the title/transcript mismatch. - **Low confidence:** Audience identities, the exact wording of audience questions, and all visual or data-backed claims. - **Attributed but unverified:** Claude's analysis outputs, specific percentages, and product capabilities (including Phoenix). Good afternoon everyone. My name is Encore. Uh I'm going to be teaching you how to make your agent cook. Again, not food related. I don't know. Uh but yeah, we're going to get into it. Um so what I mean by that is actually before we get into that, let's go ahead and start with a quick question and a poll with everyone. Um, I'm sure a lot of you are building agents or have even shipped agents. I'm just curious how many of you currently are shipping based off vibes. And when I say vibes, it's not a bad thing. I mean like, hey, you're checking your answers. You're looking at it. You're making sure that it's potentially doing the right thing just based on the input and output. You're making sure that it's correct or not. How many of you all are doing that? Okay, pretty good reasonable amount of you. And then how many of you guys actually have like set eval set up like you know like oh you have LM as a judge or codebased evals or some sort of eval setup. Okay, a little less. Not too much less though. That's pretty good actually. And then how many of you that actually have your eval setup trust it enough that you know you're good enough? You have a full coverage of your evals and you're ready to you're continuously ready to keep going and moving forward. Wow, that's that's pretty impressive. That was a solid hand up, which is kind of what I expected. I mean, it's very hard to really trust your emails because they're hard to build. You don't really know what you're looking for. It's not a deterministic use case. And it's very hard to determine what we need to actually evaluate to make sure our agents are working properly. And that's kind of one of the biggest issues is a lot of people are going to start off in a very good spot, which is based off vibes. It gets you going. It makes sure that your agent's doing what you expect it to. But realistically, you can't really ship based on this feeling of, "Yeah, it's correct." Because when you're in production, you're not going to know where things are broken, what's going wrong, unless you have actual eval set up to find out where things are breaking and how you can actually directly fix it. So before we get into eval, what I want to kind of take is a whole step back and talk about how you can like what your eval would be on top of. And that would be on top of traces. And why we need traces is if you've ever used traditional software engineering, you have something that you do call traces. It's for APM monitoring. It's making sure your systems are reliable and are working. But realistically, what you you can actually go ahead and debug those and it's very easy to do because you have a very deterministic flow, right? You can even see right here, this is very easy to follow and this is more or less what like code looks like. You have a handle submit method. You check if it's valid. If it doesn't, it errors out. If it is, then it goes down the next list and continues down that process. So, it's very easy to figure out where things are b are breaking based on a just based on that sort of uh path history. But in an agent, unfortunately, with your code, really all you have is that one line of code, which is just going to be agent.run and then the query. And ultimately, what it's going to do is kind of up to it. Will it call a tool? Will it go ahead and do some analysis? Will it do some internal thinking? What is its next best step to do? and its actual best course of action. We don't know until we actually see the traces. You won't know what your agent's doing. So that trace kind of helps you define that. And so as a kind of a decent example as to what a trace would look like would be this. You have your agent call. You go ahead and see the LM calls. You have your documents you're retrieving, your tools that you're retrieving. You basically have every step along your journey that you would see of what your agent's doing. And here you can even go ahead and see, hey, we have for example this tool.get order status. It's the wrong tool occult. The user asked us, hey, I was supposed to get a refund here. Where's my refund? If we respond back, hey, your refund was issued and they know that that was their status based on their order status. That's not what they were looking for. They were looking to understand why hasn't their refund arrived? So we need to potentially call different tools, do different actions, and there's so many different varieties even just knowing that there's an issue here. But we'll dive into that in a little bit. So when you actually go ahead and understand, okay, we have a trace. Now I want to go ahead and debug what's going wrong with my agent, we're going to start with where first you're going to identify those issues, right? Those production failures like we just did in that last slide. We found out that tool was wrong. But what's your next step? You want to go ahead and find out, okay, not only where did it go wrong, can I get a nice sample size of that sort of information that I can then use for experimentation and iteration to improve the agent when I actually want to go back and test against those cases. Then what actually was the issue, right? Sure, the tool call was wrong, but should the LM prompt change? Should we add in guardrails? Should we change up the tool? Was the tool just doing the wrong things? Right? There's so many different areas that we need to test and go against. And so we can go and test against those build out evaluations to continuously monitor those use cases. So for example, if you find out, hey, we need to make sure that we set up a guardrail here. How are we going to do that? We want to and then we want to make sure that guardrail is doing its job. So we set up a evaluator on top of that guardrail making sure is that guardrail doing its job by prohibiting certain information from leaking. Finally, you actually make those changes. run that experiment that I was talking about in the beginning and finally ship it to production and do that whole loop over and over again continuously. Now, this is your standard go-to process for actually improving your agent. It's pretty long, pretty manual, and pretty tedious. It does it does work, but we want to kind of iterate and improve that. How can we do that? We want to make sure that we're not spending hours, days finding the issue, finding the trace, finding all these different issue points, coming up with the potential solution, going through that full eight-step process manually. We want to pull all of that together and bring it into minutes. The best way to do that would be right here. We're going to pull in all the data from all our different sources into one specific area. We're going to start with, you know, pulling our ap our arise traces so you can get all the information of what your agent did. You're going to pull in your evaluations to see where things actually went wrong. You're going to bring in your annotations from any humans that could theoretically tell you a little bit more information as to, you know, where it could go right, how they can improve it. You bring in your actual APM traces which could be hey is the system up or down. you bring in your codebase so you can understand all right based on this specific area where it failed the tool call what changes can I make in my codebase so that we're c we're we're going down the right pathways we're calling the right tools we're fixing our tools that are broken and so you'll bring in all of these into one final space which is which is our coding agent whether you use cloud code cursor codeex it doesn't matter all of those you can connect all that data into and it can automatically do all of that work for for you. So, I know this may sound really complicated and I promise this also probably doesn't help to make it look simpler, but it is very simple to get it set up. It's going to be just a couple steps of just, you know, setting up your CLI and skills and you're ready to go. You get access to a bunch of information, a bunch of different tools that your agent can then do, which could be for example here, which are like, hey, you can get your code instrumented. What that means is you can get all the traces from your agent into whatever platform you choose like Arise. You could go ahead and pull down those traces, build out data sets, run experiments, run evaluations. You have the full gamut of the experience all from within your coding agent. So, what I'm going to do here is we're going to go ahead and look at a specific issue. And I want a little bit of kind of uh voices from this side of the room uh where we can get some idea of like hey what do you guys think could be the actual issue. So I'll give you a little bit of background. This agent right here is a simple financial agent. I've asked it should I invest in this company? I think I chose Tata Motors. Um anyways and ultimately it gave me a response which said yes you should. I don't think it's directly in the screenshot, but I promise you it told me that. Um, and ultimately that's not really what we want to do because I'm not registered to give any financial advice, nor is this agent, and I don't want to get in legal trouble. If you guys can help me figure out maybe, hey, what do you guys think could be the issue in this agent? So, what she said was, "Hey, maybe the agent needs to ask a lot more questions back to get a little bit more information before I can give a solid, concrete answer." That's a very good guess. Anyone else have any ideas? Okay. And how do you suggest we potentially improve that? By the way, he said it should ask, again, pretty similar. It should ask more questions before making that recommendation, but how do you suggest maybe we can go ahead and do that? \>\> More context, more asking questions, more details. \>\> Okay, so maybe some prompt changes to add in some context to get more context. One more. \>\> Sorry. \>\> Where's validation? \>\> Where's validation? Yeah, that's true. We don't have any sort of ground truth to compare against. We don't know if any of this is correct. So there's no validation that any of the answers are even correct. You're right. There could be so many different issues. And truthfully, oh yeah, go ahead. \>\> Question. Does the agent know the user's financial background? \>\> It's a great question. In this case, no, it does not. And yeah, it should definitely get a lot more information into that. Uh he asked, by the way, should the user does the does the agent know any information about the user's financial background? Um and yeah, it definitely does not. And there's a lot of areas that this agent could be fixed. And there's genuinely genuinely not one correct answer, right? But and I genuinely don't even know what the right answer is to this because this agent is still messed up right now. And we're going to go ahead and try to fix it today in this demo. Um, and so what we're going to do is we're going to go ahead and set up the CLI and skills, get that set up so we can actually get our coding agent to go ahead and understand what's going wrong with our to go ahead and pull down the traces, understand what's going wrong with our agent, look at the evaluations we already have run, look at the annotations that exist, maybe even do some of its own analysis, and potentially come back with some future suggestions just like many of you have done today for how to actually improve this agent. So that hey, maybe we could have it ask more questions. Maybe we could add a guard rail so it only responds to certain questions and doesn't give financial advice where it's not warranted. So there's lots of different ways that it can come to do those fixes. To get it started, what I'm going to do is I'm going to go ahead and pull open let's see, do I have it here? Yes. Let's go ahead and pull open a new session. Um, okay, cool. I'm in the right space. All right. So, generally what you'd want to do whenever you want to set up the CLI and skills, super easy. It would just be a couple steps to get that set up. Um I apologize. Oh, here we go. I know what I'm doing, guys. Uh here we go. So, we're going to start by setting up the CLI. You would just go ahead and copy this one line of code to get that set up. So, I'd go into back into my terminal and install that. Of course, I already had that installed right before this, but just goes just very simple and quick to set up. Then what you do is you would create your actual profile, which would help you connect it to wherever your data stored. In this case, it would be arise. And you create that profile so it can go ahead and pull in those traces and visual and get that sort of information from those evals and annotations. Once you have that profile created, you'd go ahead and set up your skills. You could do this in a multitude of ways depending on your coding agent. For today, we're using Cloud Code, so I use the marketplace to set it up, but it's pretty easy to set up however you want. From here, let's go ahead and we're just going to ask it. And again, uh just as a warning, I am using dangerously skip permissions. Please don't do this. This is not actually good practice. This is just for a demo, so I don't have to keep pressing yes. Um, but we're going to go ahead and talk to Claude and have it go ahead and take a look at these traces. So, I'm going to go ahead and see if I can find the prompt I asked earlier, which was just use this skill and pull down the traces from that specific agent. Find out what issues exist that we can determine from any, like I said earlier, right? What any issues that exist from any of the evals, any of the annotations that we've done. do your own analysis and tell me what eval maybe I can build out in the future to make sure that I don't run into any of these issues again. And so this is going to take a couple minutes. Um, in the meantime, I'm going to go ahead and first go and show you kind of how those traces look and how you can actually start triaging it if you want to do it manually a little bit. But while this is running in the background, let's go back into the actual uh run into the actual agent that has those sort of queries that we've asked. So, we'll go ahead and ask it a pretty similar question. Um, you know, tell me if I should invest into, let's say, Nvidia this time. It's a little bit more of a well-known stock. And we're going to get that and it's going to try to tell us some sort of information. And as that data comes in, looks like I've run this earlier. So, and you can see it's coming in. Um, but from an earlier run, we can kind of see what it's done already where you can see that query we asked, you can see the output that it'll finally generate, which we'll see how similar it is in a minute from this previous run. Um, and you can see all the steps that actually happened in your agent, right? So, you can understand, hey, we had a supervisor agent, a financial data agent, you can see the LLM calls, you can see all your different tool calls. you get a lot of information right away that you can start using just like we done in the initial slide to see where those issues could lie. Now, let's see. Did Claude finish up? Looks like it's still running. It's doing its thing. We'll let it cook. You know, it's kind of the point of this. Um, while it's cooking, anyone have any questions as to like, you know, how you can set up any sort form evals, any sort of traces, anything along those lines? Anyone wanted to learn a little bit more about Yeah. \>\> Great question. Yeah. So, you would go ahead. So, what it does, it actually goes ahead and downloads a lot of those traces um into the into your local system. So, what it'll do is actually let me go ahead and show you what it how it actually goes ahead and accesses the system itself. So when you give it access to these skills, what you're actually giving it access to is the CLI underneath. And what I mean by that is you're going to have all these different CLIs that it can integrate with. So you can go ahead and set up and find the data sets. You can pull down data sets. You can have it interact with the essentially interact with the UI where it can set up evaluations on the platform on your behalf. So you can monitor continuously monitor when your agents are failing, build out data sets, run experiments. It can basically do anything that you that you would be able to do in the platform. Sometimes it'll download, sometimes it'll make API requests, but that's generally the uh idea here. Great question. Any other questions in regards to CLI tracing? Okay, let's go ahead and see what Claude's done. Oh, still crunching away. Um, so as we can see here, let's let's walk through what Claude's done so far. So, we can see that we've gone ahead and gotten the spans. It's digging into the evaluations to figure out where those issues are. Um, it looks like it's not quite there's definitely found one very big issue with some numeric groundedness. I know someone mentioned that earlier as a potential issue or something related to that. Um, we're also seeing some tool calls directly fail. Um, and let's see. We also see, yeah, so we're getting a lot of different errors. Oh, looks like it just finished up. And so you can see here we get a full list as to like, hey, what's going wrong with our agent and what we can do to kind of fix it. So while it's finishing up that, you can see here one of the biggest alarming things right out of the gate is about 44% of our agent interactions are resulting in a bad summary. That's really bad. I did not know that. So, we definitely need to fix that one. Another big issue is Oh, did not mean to scroll there. Another big issue is, hey, we're also just getting a lot of fabricated answers, which is just as bad as this the summarization being wrong. So, it sounds like it may be related, it may not be. We can kind of debug that. We're also seeing a relatively high rate of us just redundantly calling tools. It's a waste of our time, our resources, and our money. So clearly, we're doing a lot of bad things that we need to go ahead and fix. And that's just from the eval from Claude's own analysis. It's able to even go ahead and figure out, hey, there's other issues that are coming along. We're getting silent tool failures. It's telling us it succeeded, but ultimately it's not giving us any actual responses. We're getting the agent hallucinates when tool fails instead of disclosing it. So again related but hey that means maybe we need to not just do uh you know some eval built on top of this to understand when the tool is failing but we also need to make some prompt based changes to make sure that the the agent doesn't hallucinate here and you have a multitude of like different things that it'll go ahead and tell you as well as the evals it can build out. So what we could do is we can actually go ahead and tell it, hey, let's go ahead and build number one, which was the tool success silent failure eval, and fix numeric groundedness so that it'll make sure that whenever it does respond, it's always grounded to some sort of actual reference point and it's not making up these numbers. But we're also going to head and make sure you test this against a data set uh against the spans that caused the failure initially using and then we're going to go ahead and use another Arise skill which would be here the Arise uh data set skill, the Arise Oh, there we go. Arise um evaluator skill as well as the Arise experiment skill. So here it's going to be able to let me just go ahead and run let this run and build out evals. All right. So again I'm being pretty vague here but essentially I'm just telling it go ahead and make the changes that you recommended as well as go ahead and run these experiments for us. So now we can actually compare against where it failed to make sure it passes against that right and so in this scenario we can see okay it's going to go through and take its time. Um I'm going to go ahead and show you a previous run because again this will take some time and while we want to let it cook I don't want to have you guys staring at a screen where Claude's just discombobulating or whatever it does in its free time. Um, so let's go ahead and look at where I've actually gone ahead and asked Alex or sorry Cloud Code to go ahead and answer some of our previous issue that I ran into. And one of those was related to hey, I've had some times where I'm trying to get some answers about uh stocks that aren't necessarily from the United States. And so we're having errors where we're talking about like South Korea or looks like they're all South Korea based, but let's assume there's other answers here. Maybe Japan or China and whatever, right? And you want to see and you want to be able to understand, hey, I want to be able to compare stocks across different markets. And clearly people are asking that, but we don't support it. And so what Claude did, it went ahead found those areas where that issue existed. Like I said, it looks like it was all South Korea here, but it found those areas. It built out a data set for me, which I can then go ahead and test against to make sure it works. And then it went ahead and ran experiments. And it made a couple changes. So, it started with the initial one, which was the baseline, which was buggy, which as we expected would result in 0% uh passing rate, which is what happened when we initially got the actual productionbased traces. It also made a quick fix where it normalized it by adding in that uh country name. It didn't work. Why? Because the tool I was calling was also outdated and I needed to update to the stable version. So then it made a third iteration. And you can see that on this chart here as well where we were able to see, hey, the first iteration didn't work. It went back, made a fix, and finally got us to the final iteration where we're getting about a 90% success rate. Clearly still not 100%. We still need to make some changes, but hey, we're doing a lot better than we were earlier. So, this is like a great way to get you going so you can actually start from, hey, I know there's an issue to here's a fix. Now, I know we have quite a bit more time. I want to check in. How is everyone doing? How do you guys feel? Do you guys have any questions about how you can go ahead and get this started? Yeah. Great question. So his question was how where do the experiments actually run? Where do the evaluations run? Where does everything run? Um so you do have a little bit of customization there. In this scenario, uh what I did is I had the experiment run on my local machine, which is I think what will be the case most of the time. And then what you what I did afterwards was I went or you can host it on a cloud or wherever you'd like to experiment on. And then you can go ahead and actually run the evaluations on Arise itself. So in this case, I built out LM as a judge, which would then go ahead and let's see if I can see the evaluator here itself. So you can see we built out two different evaluators, a codebased and an LLM based. I actually had Claude go ahead and do this. It built it out for me in the platform. And this was the final LLM as a judge because it actually kind of failed in getting the codebased data to set up. So that's why we have two. And so you can see the LM as a judge. It went ahead and built it out for us with this prompt and got it going so that we can actually compare. Hey, did the agent work properly? Any other questions? That was a really good question by the way. \>\> Yeah. \>\> Very very good question. Um so eval can be set up however you like. We have of course people can people can definitely use whatever sort of models they want. No one wants to be limited to just claude or just open AAI especially in today's modern age where you can see AI models literally ship every other day and then you want to be like okay I want to use that one now right and also some people have a preference for slightly cheaper models some people have a preference for making sure their evals are topofthe line because ultimately they know their agent's only going to be as good as their evals so they use the highest end models for their evals even if they're not necessarily using those in production so it kind of varies you have full customizable control over cut. And so what you can see here for example, you have a full list of whatever sort of model provider you want to use and you can connect those even if you want to use any sort of local um open source models as well. Yeah, good question. So her question was if we have if I spin up a smaller agent to go ahead and make some of these bug fixes, how does this compare to that? Realistically, you don't have to use cloud codeex. You can use whatever sort of agent you want. The benefit of what we've provided is a skill that you're able to go ahead and pull down the information from your traces to make those fixes. So, you're not making those fixes just based on, hey, I know this is wrong and the output's incorrect. You're making those fixes based on, hey, I know this is wrong. Here's where it failed. Here's why it failed. Here's how we can go ahead and fix it. because you have not just the information of your codebase, you have the information of any of your other APM tickets, whatever or APM logs, any of your other tickets that exist in your Jira systems, etc. as well as your actual arise traces, which would be like here's the pathway the agent took, here's where it messed up, your eval, your annotations, etc. Any other questions? Yes. \>\> Are these features available in Phoenix? \>\> Great question. Yes. Uh we do have so he asked is are these features available in Phoenix. Phoenix is Aris's open- source version. We do have I believe skills and MCP as well in Phoenix which you can go ahead and hook up to your agent and it can go ahead and pull down the same sort of information. Of course there are going to be some nice to haves in the enterprise edition where you have access to these online evals in a certain way that are much easier to format and set up. You have a little bit more dashboarding and a few other features. But yes, the the skills in CLI you can connect to your coding agent. \>\> Yes. \>\> Agent said that summary. So how would you classify something as like a bad agent know that's a good summary or a bad? \>\> Very good question. So her question was in what I showed earlier there was a eval or there was a response from Claude that said hey there was a bad summary. How do we know what is a bad summary or a good summary? In that case, what I actually had set up, it was actually based on one of the evals that I had previously set up. And so, we can actually go ahead and look at that evaluator here. We can go ahead and look at let's see some I actually don't remember which one, but I'm going to go ahead and pull open this one and hope this is the right one. Um, so as you can see here, this is basically the summary eval that I'd created. It's basically taking a look at the information we've gathered throughout our entire agent, all the sort of context we've given to that summary agent and making sure that it's referencing the right points, it's answering the question accordingly, etc. Um, and so that's how we've gotten to determine whether or not it was a good summary or not. \>\> Uh, what if we set up a skill in the first place to see if it can do a good summary? give it given us a good summary all the time just like we set up a skill instead instead of running. \>\> Yeah, great question. So, she's asking why can't we just set up skills that you know ensure that it gives a good summary and yes absolutely that is what you want to do. You want to make sure that you update your prompts, update your skills, update your coding workflows so that your agents will give the best summaries. The emails are only there to help you determine where there are issues, monitor those issues, make sure they don't repeat, right? And so you set up that skill, great, it works for the next two months, two months later because there's a model regression because people start asking questions in a different way. You end up getting different results and you don't want your e you don't want your performance to degrade because of that. So you want to be able to monitor and observe that. That's the point of the evals. Did that help? Beautiful. Um I don't know for how much time we have left but go ahead \>\> later agents are assessed and soc. So your question was if I've run into issues with linting with the agents \>\> agents seems like more \>\> good question. Yeah. So he's basically saying hey um have you seen the issue where hey you've given access claude to all these skills and all these sort of traces and all this sort of information but ultimately most of the time I've seen with cloud code or any of these other sort of coding agents they end up focusing on linting rather than any other sort of issue am I correct yeah that's a very good understanding and that's kind of why we built out these skills which kind of focus on ensuring that they hyperfixate on the actual issues that have been given to them, right? So they'll get all these trace information so they'll understand, hey, it's not going to be a linting based issue because you can see that it's calling the wrong tool and that's not going to happen because of linting and it does have some reasoning capability to understand that. Um, and so genuinely I haven't seen that so much when using the skills. Awesome. So as it Oh yeah. Does does the system make it easy to understand? Sorry. \>\> Great question. Yeah. So, how so his question was, does it make it easy to understand how much it costs to run those evals? Absolutely. So, one of the best parts of running evals is in the platform is you're going to be able to get full trace history of what your emails did, which also comes with the added benefit of, you can see right here, a nice little cost. And so, you'll be able to understand how much your emails are actually costing you because yes, of course, anything anytime you're running another LLM, it's going to be additional cost to you. So, you want to make sure it's not. And of course, what's really nice about building out evals, whether it's in the UI or via the coding agent, you can have those set up basic requirements like, hey, we want to sample specific amounts so we don't run it across 100% of our traces and 100% of our calls. Um, you can make it focus on specific areas. And of course, you can always dictate that in the coding agent or on the platform itself. Um, so as a kind of refresher as to everything we kind of covered today, again, it that's the wrong one. Oh, I think I have it over here. Um, so essentially what I really want to give you as a takeaway, these AI agents, they're great. They will do basically everything you ask them to, but again, they're not magic. Like he said, they can definitely focus on linting. They can definitely get caught up in small little areas, but the best part of them is they will automate your full workflow from beginning to end. So whether it's that starting from observing where the issues lie, finding those issues, being able to actually pull it down, understand where the issues lie, create those data sets, build out exact fixes, test them to make sure that they're working. And finally, you can then go ahead and be the manual human to make sure that hey, it didn't just go back and fix the linting changes, right? And we can make sure that it actually made changes that actually help improve the agent and not go for degradation. That's kind of the general gist of what I wanted to show today. Um, any other questions on this before, you know, I can maybe show one last thing of seeing. Oh, still finishing up what I was trying to do earlier. I was hoping it might be done after this. took about 14 minutes. But um while that's finishing up, any other questions on how we can use our agents to automate our workflows and improving your agents? Yeah. \>\> Great question. So his question was, are we primarily relying on Claude or any of these coding agents to actually go ahead and give us the evaluation if things are working well or not? Um, yes, you can definitely rely on Claude to build out everything. You can also manually do it. You can also ask Claude to do the specific evaluations you want. So in this case, when you saw that nice clean table, that was actually defined by the evaluations me and my team had set up prior to this. Um, we'd gone ahead and looked at, hey, these are specific areas that we know are potential limitations in most agents that work in this workflow and we set up those as generic ones, which is why we got a bad summary. We had some, hey, is the agent healthy? Is it giving the right are we getting tool calls correct? Right? We know those are general areas, but what Claude was able to do was look at our actual traces, understand based on those traces, what are the actual issues we're falling into today. Right? we have our live agent in production. I'm not going to be able to go through all of those manually and find those, but Claude can or Claude cursor codeex, whatever, right? And that's where like one of the biggest superpowers that you're able to give these coding agents to be able to do that for you. I saw another question here. Good question. So his question was is there any sort of benchmark data set that we can see or any sort of like visualization that can tell us hey this is a higher quality LLM to use for evaluations more expensive less expensive um realistically no \[laughter\] that would be sort of something you'd have to do for your own analysis you'd have to figure it out because ultimately one of the things that I like to say is if you're going to use out of the box eval to judge your agent you're going to get out of the box results and that's not what you want when you're building customtailored AI agents which are built to be personal for you. Um, and the same goes for benchmarks, right? Benchmarks are great. They'll show you, hey, this this is a good agent. This is a good sorry, this is a good LLM. But realistically, what you want is, is it a good LLM for me? And that's going to be tailored to your use case. So, if you're building for finance, maybe you'll use a specific one. If you're building for, you know, law in finance, it might be even this slightly different model, right? So, it really depends on what you're building it for. Cool. I think that's more or less it more or less it from me. Um, if anyone has any other questions, I'm happy to answer them. But yeah, thank you guys.