Talk by Dat Ngo of Arize AI on observability, evaluation, and experimentation for LLM applications. Covers what observability means for nondeterministic systems whose execution paths vary across runs, five types of eval signal (LLM-as-judge, human feedback, golden datasets, deterministic checks, business metrics), and the scopes at which evals can run (single span, multispan, trajectory, session). Discusses the open-source Arize Phoenix tracing platform, which runs as a single container without Kubernetes, and the enterprise AI layer Alex, which scans traces and auto-generates evals. # Channel Presentation Observations — Video: LLM Observability, Evaluation, Experimentation Platform (Dat Ngo, Arize) ## Speaker & Affiliation - The speaker self-identifies as **Dat**, an **AI architect at Arize AI**: > "My name is Dat. I work at Arize AI, so I'll talk a little bit about what that is." - He describes his role as working with **large enterprise customers**: > "I work with a lot of the largest enterprises across the world." ## Intended Audience The talk is aimed at a **technical, developer/AI-engineer audience** with hands-on agent-building experience: - The speaker polls the room: *"who's built agents, who's building agents, who's productionized agents, who's building harnesses"* and concludes *"we're all pretty cracked, so that's good."* - He references concepts like **OpenTelemetry**, **auto-instrumenters**, **traces and spans**, and **CLI/tool/skill primitives**, assuming audience familiarity with engineering terminology. - He also acknowledges a second persona — **non-technical domain experts and product managers** — as a user class Arize's platform serves: > "We allow folks to be able to run evals just in a non-technical way." ## Claimed Scope of the Channel/Company Explicit self-descriptions of what Arize does: - **Three-pillar framing**: *"observability, evaluation, experimentation, but really it's just how do you make AI work."* - **Enterprise vantage point claim**: *"We get to see what every team is building, how they're building it, what are the biggest pains that they face."* - **OpenTelemetry-first positioning**: *"everything we do is through open telemetry… being Otel first, we're really prepped for a lot of these use cases."* - **Two-product structure**: - **Arize Phoenix** — open source, *"single container… deploy it locally… doesn't require Kubernetes."* - **Arize AX** — enterprise tier, used by companies he names as **Uber, Booking, and Reddit**. - **Strategic direction claim**: > "Our ultimate goal as a company is actually to automate you out of this process… We think the whole flywheel is very much automatable." ## Observable Presentation Patterns - **Demo-driven format**: The talk includes live product walkthroughs (light/dark mode switches, showing traces, distributional agent views, eval templates), suggesting a conference-style presentation with product demonstrations. - **Educational framing over pure sales pitch**: Concepts are explained from first principles (e.g., evals as *"how do I derive signal from my systems"*, span vs. multi-span vs. trajectory vs. session evals). - **Positioning as expert insider**: Credibility markers include token usage ("somewhere between 100 billion and 1 trillion tokens last year") and referencing a recent **Anthropic paper on managed agents** "that came out 2 days ago." - **Philosophy of demystification**: A recurring theme distinguishing claims about AI engineering: > "It feels like magic, but it's not magic… It's all just engineering." - **Trend-aware product positioning**: The speaker frames Arize's tooling around developer workflows like **Claude Code and Codex**: > "We realize people are very comfortable with their Claude Codes and Codexes. So we expose all the primitives via the CLI and a set of tools and skills." ## Notes on Claim vs. Demonstration - Customer names (Uber, Booking, Reddit) and enterprise reach are **creator claims**, not independently verified in the material. - Product capabilities (Arize AX distributional views, eval templates, "Alex" AI assistant) are **demonstrated or described live**, but the demonstrated content is limited to what the speaker narrates during the talk. - No curated metadata was supplied; all observations derive from the title and transcript. # Delivery Observation: Dat Ngo, Arize ## Tone & Overall Style - **Conversational, casual register** throughout: "I do like to vibe it out," "we're all pretty cracked, so that's good," "Whatever." - **Self-positioning as a practitioner-insider**: leans on personal experience ("I do spend a lot of tokens," "somewhere between 100 billion and 1 trillion tokens last year") rather than formal credentials. - **Demystifying framing** as a recurring rhetorical move: "it feels like magic, but it's not magic, right? It's all just engineering." Repeated later: "Meaning it's not magic again, but it should feel like magic." ## Vocabulary & Signature Phrases - **"In a nutshell"** — appears at least three times as a summarizing device: - > "So, if I had to really distill down, you know, what we do in kind of a nutshell" - > "And that's that's observability in a nutshell." - > "TLDR, you know, that's that's observability and evals in a nutshell." - **"Right?"** — extremely frequent tag question used to check shared understanding ("just a different flavor coming out... but it's not magic, right?"). - **Repeated fillers "you know," "like," "hey"** — e.g., "you'll say, 'Hey, my can I get my LLM to approximate this this thing.'" Also frequent immediate word repetition ("this this," "the the evals"). - **Flavor/scope vocabulary**: "five flavors of signal," "what flavor of evals do we want to run, but at what scope and depth?" ## Teaching Habits & Structure - **Explicit three-part roadmap**, stated and then followed: observability → evals → experimentation/improvement ("really what we're going to cover today is just three things"). - **Rhetorical definition pattern** — defines terms via embedded question: "observability, which answers the question of like what's happening in the thing that I've built"; "Evals are just simply how do I derive signal from my systems." - **Audience interaction**: "Maybe by show of hands, like who's who's built agents... who has no idea what a harness is." - **Quotable hypothetical dialogue** to explain features: "you can just simply say like hey, do you see any issues with my application?" - **Concrete micro-examples** to ground abstractions, e.g., JSON validation: "Does it have the schema? Does it have, you know, these fields that are non-null?" - **References to current external material**: "I don't know if you any of you read the Anthropic paper, managed agents that came out 2 days ago." ## Live-Demo Behavior - **On-the-fly adjustment to visibility**: repeatedly switches display modes mid-demo — "I realize this isn't the easiest to see, so let me change over to light mode... sorry about that," and later "let me turn this light mode." - **Minor live-navigation stumbles**: "oops, let me close this," "Let me kind of come here." - **Humor about product defaults**: "You guys can't tell we love dark mode, so we probably should try to go light mode." ## Product-Pitch Weave - Teaching and marketing are interleaved: technical explanation of traces/spans flows directly into Arize AX capabilities, closing with a two-product split (open-source "Arize Phoenix" vs. enterprise "Arize AX"). ## Uncertainty Notes - Patterns like "right?", "you know," "in a nutshell," and the definitional question format are well-supported by multiple examples above. Single-instance moments (dark-mode joke, hand-raising) are observed once and may not be stable habits — they're reported as isolated incidents. # Observation Record: LLM Observability, Evaluation, Experimentation Platform **Video:** LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize **Speaker:** Dat Ngo (self-identified), AI architect at Arize AI ## Speaker Background & Framing - Works with "the largest enterprises across the world" on observability, evaluation, and experimentation — summarized as "how do you make AI work." - Claims heavy personal usage: "somewhere between 100 billion and 1 trillion tokens last year" (at OpenAI DevDay). - Claims a unique vantage point: seeing "what every team is building... what are the biggest pains." ## Core Theses ### 1. AI is "software reimagined" > "It really just feels like software reimagined. It's really the same set of patterns, just maybe a different flavor... it feels like magic, but it's not magic, right? It's all just engineering." ### 2. Three pillars The talk is organized around: **observability**, **evals**, and **experimentation/improvement**. - **Warning on regressions:** fixing one issue in a non-deterministic system "might have actually produced like two or three regressions that you didn't really know about." ## Observability - **OpenTelemetry-first:** "everything we do is through open telemetry," enabling auto-instrumentation with one line of code. - **Traces/spans as audit record:** "code doesn't audit agents or harnesses, it's actually the telemetry that does that." - **Sessions:** back-and-forth conversations/state; references the Anthropic "managed agents" paper released "2 days ago" as "a pretty awesome read." - **Distributional views** (Arize AX): see all paths, branches, loops across agent instantiations — e.g., percentage of traffic per branch, latency attribution by component. - **Trajectory analysis:** example root-cause — components called out of order ("I did B before A and actually B has a dependency on A"), fixed by adding context/instructions. - Analytics/dashboards "aren't dead" — enterprises want real-time custom views. ## Evals ### Five "flavors" of signal - **LLM-as-a-judge** — "may seem like a simple concept, but in actuality it can actually get quite complex." - **Human signal** — end-user feedback, valuable to technical and non-technical roles alike. - **Golden data sets** — trusted domain-expert labels; used to tune LLM-as-a-judge ("can I get my LLM to approximate this... data set that I trust?"). - **Deterministic/logic-based evals** — e.g., valid JSON, schema conformance, non-null fields — "you don't always have to use an LLM call or even humans." - **Business metrics** — "make more money, save money, or save time." ### Eval scopes - **Span eval** — single input/output component. - **Multi-span eval** — data across multiple components (e.g., "how well are agents passing data back and forth"). - **Trajectory eval** — did things get called in the right order to finish the business process. - **Session-level eval** — evaluate the state machine, e.g., "was the user ever frustrated? Did we answer all of their questions?" ### Qualification > "Just because you can eval something doesn't mean you always should... what are the minimal set of evals I can get away with... because there's a cost associated with this stuff." ### Two personas converging - **Technical users** (AI engineers): good at building, automating, frameworking. - **Domain experts / PMs**: know what the AI experience should be; should own prompt engineering and eval selection. - "You want people who can code coding and you want people who know the domain to work in that domain." ## Experimentation & Improvement - Improvement = changes: to **prompts, models, orchestration, configurations**. - Workflow: find bad signal in traces (or upload input/output datasets), collect into datasets, run experiments — via UI or programmatically. - **All primitives exposed via CLI / tools / skills** for coding agents (Claude Code, Codex). - **"Alex"** — Arize's built-in AI agent that can plan and run tasks (e.g., "do you see any issues with my application?"). ## Company Direction & Positioning - **Goal: "automate you out of this process"** — the observability/evals/experimentation flywheel is "very much automatable"; "it's not magic again, but it should feel like magic." - **Belief:** users shouldn't even have to choose their evals — AI should create evals on the fly from trace context. - **Direction stated as belief, not present fact:** "we are very heavy believers." - **Products:** - **Arize Phoenix** — open source, single container, local deployment, no Kubernetes required; for "engineering first folks." - **Arize AX** — for largest enterprises; speaker names **Uber, Booking, Reddit** as customers. ## Preserved Tensions / Qualifications - Automation framing is aspirational; much of the demo was "the nice manual way." - Speaker's token-usage and industry-vantage claims are self-reported, unverified. - Enterprise named customers stated as fact by speaker (promotional context). # Video Attribution Record ## Video - **Title:** LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize - **Video ID:** `01a15ca9-0f8e-75b6-9366-08115d0585bd` ## Speakers and Roles | Speaker | Role | Notes | |---|---|---| | **Dat Ngo** | **Creator / Presenter** | Employee at Arize AI; identifies as an AI architect. Delivers the entire talk. | | **Audience (unidentified)** | Audience | Participates via show-of-hands interaction; no speaking lines attributed. | - No **narrator**, **guest**, or other identified speakers present. ## Speaker Self-Description (Provenance) - Dat Ngo states he works at Arize AI and works with "the largest enterprises across the world" on observability, evaluation, and experimentation. - Claims attendance at the "last OpenAI DevDay" and estimates personal usage of "between 100 billion and 1 trillion tokens last year." ## Referenced Sources and Third-Party Claims - **Anthropic paper ("managed agents"):** Referenced as published "2 days ago" before the talk; described by the speaker as "a pretty awesome read." (Title as stated in transcript; not verified.) - **Named enterprise customers:** Uber, Booking, Reddit — cited by the speaker as users of Arize AX. - **OpenAI DevDay** — referenced as context for personal token usage. - **OpenTelemetry (Otel):** Cited as the underlying standard for the platform's instrumentation. ## Promotional Segments ⚠️ The talk is a **company presentation**; the following portions are promotional in nature: - Repeated promotion of **Arize AX** features (distributional agent views, evals UI, CLI/tools/skills, "Alex" AI assistant). - Promotion of **Arize Phoenix** (open source, single-container, local deployment). - The stated company goal: "our ultimate goal as a company is actually to automate you out of this process" — positioning statement for Arize's roadmap. - Claims of unique industry vantage point ("we get to see what every team is building"). ## Products Mentioned - **Arize AX** — enterprise product. - **Arize Phoenix** — open-source product. - **Alex** — AI assistant within the Arize system (speaker-named). - Third-party tools mentioned as user context: **Claude Code** (transcribed as "cloud code") and **Codex** (transcribed as "code X"). ## Comment Availability - **No comment availability information** is present in the source material. ## Uncertainty Notes - Audience members are not individually identified; only collective interaction is recorded. - The Anthropic paper title is given as "managed agents" in the transcript; exact title is uncertain. \>\> Okay, I didn't know if there's like a cut scene, but okay, so really nice to meet you all. My name is Dat. I work at Arize AI, so I'll talk a little bit about what that is. A little bit about me and kind of what I want to share today is um you know, I I work very deeply in in the space. I'm an AI architect. I work with a lot of the largest enterprises across the world to talk about, you know, we work on things like observability, evaluation, experimentation, but really it's just how do you make AI work, right? So, I do spend a lot of tokens um in this space. So, this is last OpenAI DevDay. I think I made it to probably somewhere between 100 billion and 1 trillion tokens last year. So, I do I do like to vibe it out. But I do know the space really, really well. We work with some of the world's largest companies and enterprises. So, we get to see their transformation into this space. And really what I wanted to share today was like what do I see in in the industry? So, I think we have a very unique vantage point being the company that we are. So, we get to see what every team is building, how they're building it, what are the biggest pains that they face, and really how they're trying to fix those those things. So, if I had to really distill down, you know, what we do in kind of a nutshell, it's really these three things. Maybe by show of hands, like who's who's built agents, who's building agents, who's productionized agents, who's building harnesses, and who has no idea what a harness is. Okay, we're all we're all pretty okay, all pretty cracked, so that's good. You know, I think it's really funny is that like the AI space, it really just feels like software reimagined. It's really the same set of patterns, just maybe a different flavor coming out. And it's really it feels like magic, but it's not magic, right? It's all just engineering. So, really what we're going to cover today is just three things. The first one is observability, which answers the question of like what's happening in the thing that I've built and what does that look like in in whether it's a harness or an agent. Then we'll go into evals. Evals are just simply how do I derive signal from my systems in some form or fashion, right? And then as we talk about how we make improvements in this new non-deterministic world, you'll come to find out that when you make you know, what you perceive as a fix and you fix the thing that you thought you fixed, you might have actually produced like two or three regressions that you didn't really know about. And so, like in our world, we talk about observability, right? Everything that we do, something that we're super proud about is Otel is a really strong pattern for those in the engineering space, but everything we we do is through open telemetry. So, you know, it doesn't really matter, you know, what particular type of you know, harness agent model setup that you have. The good news is that, you know, being Otel first, we're really prepped for for a lot of these use cases. So, whether it's an auto instrumenter, if you know what that is, but basically you add one line of code, that one line of code will, you know, basically see what's happening around in that particular framework or SDK, create open telemetry traces and spans, and produce, you know, these views kind of here. So, if you've ever seen a trace or span, it's basically the audit record of what did my agent do? Cuz now we know that code doesn't audit agents or harnesses, it's actually the telemetry that does that. So, traces is a big fundamental part of observability. Now, there are many other different parts of observability that you should be thinking about outside of traces and spans. You can think about sessions, too. So, I don't know if you any of you read the Anthropic paper, managed agents that came out 2 days ago. Pretty awesome read. But sessions is another one about like state. So, what are the back and forth maybe conversations, what are the back and forth states that are happening between you can think runs. For a lot of people in the enterprise, they may want to end up running I realize this isn't the easiest to see, so let me change over to light mode. A lot of folks yeah, sorry about that. Yeah. A lot of folks would like to, you know, understand like, "Hey, what is the back and forth conversations?" So, that will be something like a session. So, it's like, "Hey, what are those back and forth conversations that are that are being had?" Great. Now, people like to eval those. So, in the enterprise, you'll see a lot of folks being like, "I don't really care the deep level, you know, the agent did this tool call, that tool call." They may not care about that as much as like, "Hey, was the end user satisfied? Were all their questions kind of answered?" Now, one unique thing, I'll change this over to light mode, too. One unique thing about what we do here at Arize, this is Arize AX, is that sometimes when you think about your agent as not a a non-deterministic call, right? You want to be able to see, "Hey, what did my agent do?" For example, there's different paths that your agent could take, right? So, you know, different branches. But what if you wanted to like look over all instantiations of your agent and get a more distributional view of what's happening? These are kind of like views into like the distribution of your agent. So, what are all the possible paths and branches, also loops? It allows you to answer questions like, "What percentage of my traffic goes down one branch versus another, right? Was there a particular component in that particular, you know, branch that we took that caused whatever, a significant amount of latency?" When we start to talk about agents or different paths, you may think about trajectory evals. So, trajectory could be like, "Hey, I went down this one path and everything was really good, but for some reason when I go down this path, the the evals or the signal that I'm collecting is dropping. Why? What's the root cause? Oh, the root cause issue was that these two components are actually out of order. I did B before A and actually B has a dependency on A. So, actually like it turns out like the way my LLM decided to call these things was was mismatched, right? We need to put some context in there to say, "Hey, actually before you do this, you need to do to do that." So, there's many different, you know, views into observability. Of course, things like analytics aren't dead, either. So, you know, what a lot of the you know, folks in in the enterprise end up looking at is just they just want to build views on what their what their agents look like in real time. And so, being able to customize and build those views out, super, super helpful. And that's that's observability in a nutshell. Is like, can I see all the different layers? We have many, many different types of of layers here. Um You know, the next thing is like, okay, so I have observability, that's kind of step one. It's the same thing that happened in software, right? Now, you have to determine signal, right? And so, signal comes in actually many various forms as well. The way I like to break it down is is kind of these five flavors of signal. Um I think everyone here in the room has heard of LLM as a judge. Um and so, you know, it may seem like a simple concept, but in actuality, it can actually get quite complex and we'll kind of go through all of that. Now, you can't forget about your humans. When you think about humans, whether it's the end users using your product, it's extremely valuable signal. So, whether you're product manager or, you know, someone technical or non-technical, you do care about this signal. We've all heard of golden data sets. They're extremely valuable because it's if the third column here represents quality, you know, you trust the person who who labeled this data because they know the domain. Then you'll, you know, you'll run techniques like, "Hey, I'm going to run my LLM as a judge on some, you know, golden data set so that you can tune your LLM as a judge." You basically say, "Hey, my can I get my LLM to approximate this this thing or this person or this data set that I trust?" And then of course, we're all thinking about costs, as well. So, when we think about costs, you don't always have to use an LLM call or even humans. Determinism is super nice, so think about logic or deterministic based evals. If I go from paragraph to, you know, JSON payload, does this JSON is it a valid JSON? Does it have the schema? Does it have, you know, these fields that are non-null? And then of course, we're all building these things for some, you know, for one of three purposes, I think. So, the business metrics you care about are either some form of like, "How do I make more money? How do I save money? Or how do I save time, right?" And so, what you'll notice as you start to build really good AI products, um you'll start to have two types of personas that kind of end up coming together. So, obviously, you have your technical users, right? These are your AI engineers, your your developers of the world. These people are extremely good at building and automating things, right? They're good at frameworking. But then you have folks who who are maybe less technical, but they understand what the AI experience should be, right? These are like the subject matter experts, the product managers of the world. These folks end up, you know, you want to relegate kind of the work of like, "Hey, this is how the prompt engineering should go. Here's the \[clears throat\] evals that I care about." Because you want people who can code coding and you want people who know the domain to to work in that domain. And so, in our world, what that looks like is something like this. You know, we allow folks to be able to run evals at just in a non-technical way. Of course, if you are technical, you can attach your evals and and run them kind of programmatically if you want. But in our world, we want to be able to say, "Hey, you know, I want to be able to allow a user to be able to select, you know, their their model, be able to run some out-of-the-box template or customize some eval here." And when we talk about complexity on the eval side, right? Imagine for a second that you have built some some application, some agent, some harness. That harness is got components in it. They may be called deterministically or non-deterministically. Whatever. So evals can be run on you can think a single kind of component. You know, we call that a span kind of eval. Let me kind of come here. So the scope would be one single input and and output. I'll pull up like a more complex view of this, but oops, let me close this. But you can think of the simple span input and output as um hey, I want to look at the input and output of one part of an LLM call. So that's most people understand that and that's really really simple. Now um we also have like multi-span evals. So we can think of that as like hey, in order to run the eval that I want, it actually requires data across many different components in in the system. So if I want to say hey, how well are agents passing data back and forth to each other? Well, it turns out I need the data from every single agent and how they pass data. So that's a multi-span eval and it allows you to run more complexity. If you want to look over all of the spans in in total, that's something like a trajectory eval. Did we call things in the right trajectory to to finish the business process? Um and then there's that session level eval, right? It's it's like zooming out and saying hey, what does the state machine like if I want to evaluate that state machine of um hey, let me turn this light mode. Hey, in this conversation, was the user ever frustrated? Did we answer all of their questions? So think of that as I want to evaluate the state machine. So as you're thinking about evals, it's not generally, you know, it's it's also like hey, what flavor of evals do we want to run, but at what scope and depth? So you can get very granular and then you can also zoom out. And just because you can eval something doesn't mean you always should. It's not this exhaustive thing. You want to see like hey, what are the minimal set of evals I can get away with to understand signal of like is my application working as intended because there's a cost associated with this stuff, right? And so, you know, TLDR, you know, that's that's observability and evals in a nutshell. We'll talk about experimentation and improvement. So not everyone starts with with traces. If you do start with traces, you can take them and do really cool things like um say like hey, show me where some signal is, you know, whatever, bad. Where am I missing stuff, right? Then you can find those things, collect them up into a data set. Also, if you don't have traces, you can just upload a data set outright, input-output pairs. And then from here, you can do things like grab that data set. For example, which is just rows and columns of data. Then you can start to run experiments. Experiments can be changes. So as you think about how do I make things better for my agent or harness, it's generally changes. Changes to prompts, changes to models, changes to orchestration, changes to configurations. Think think that way. Um we allow folks to be able to test these things in a UI or programmatically. But you know, one thing I always like to share with our customers is like where is the space going? What we quickly realized at Arize here is that like most people don't want to live in dashboards or buttons or manual things. And we we very much recognize this. As you think about where the future is going, software will compress. It's going to be easier to build, easier to customize. So everything I just showed you was just like the the nice manual way to see it, but everything we've done, we've allowed you to be able to do this program like through your coding agent. We realize people are very comfortable with their cloud code and code X's. So we expose all the primitives via the CLI and and a set of tools and skills. So that's kind of opinionated. And then there's also an AI system built into all of this too, meaning cloud code, your AI system can end up calling our system. So that you can do things like you know, if you don't want to just figure out these things on your own. We believe a lot of this stuff is is can be automated. So I can go here and ask Alex, obviously cloud code or something outside of the system can call Alex. And you can just simply say like hey, do you see any issues with my application? Um and you know, because we have all the data, because we have the the hooks and everything else, you know, Alex will go in and plan and run these these tasks. So our ultimate goal as a company is actually to automate you out of this process. Observability, evals, experimentation and improvement. We think the whole flywheel is very much automatable. Meaning it's not magic again, but it should feel like magic. So our main goal is like one day you you you work with Arize and you pull this, you know, you pull the ecosystem down and then it just works. We are very heavy believers that you shouldn't even have to choose your evals. Like an AI should have context of like hey, here's the traces, here's what's happening. Let me create evals on the fly and and think about them for you or like hey, something has changed. I know I need a new eval. But you notice Alex is already getting to work with like hey, what's happening here? Looks like we have some high latency. We have, you know, some errors detected, things like that. Um So in a nutshell, this is kind of where we're going and and what we're after. And so as we think about, you know, this this world we actually have two products out today. So for for more of the engineering first folks, we have Arize Phoenix, which is open source. The really nice thing about Phoenix is it's single single container. You can deploy it locally. Doesn't require Kubernetes layer. And then for our largest enterprises, they use Arize AX. Which is kind of generally reserved for um you know, some of the largest enterprises between like Uber and and Booking and and Reddit. You guys can't tell we love dark mode, so we probably should try to go light mode in some stuff, but yeah, in in a nutshell, that's that's kind of what we do and and who we are and you know, if you guys want to chat about anything past that, super excited, but yeah, thank you very much for for your time today. \>\> \[applause\]