Talk by Hossein Niazmandi (Braintrust) on why building an eval platform is harder than it appears. Covers the two pillars of agent quality — evals before production and observability after — and why non-deterministic LLMs require both. Walks through stages of eval maturity: spreadsheets, custom UI, side-by-side experiments for PMs and domain experts, and a flywheel turning production failures into test cases. Discusses the data challenges of agent traces (large, nested JSON needing real-time and long-running queries) and Braintrust's query language BTQL, closing with coding agents running evals themselves and surfacing unexpected failures while humans make final decisions. Recorded at the AI Engineer World's Fair 2026 in San Francisco. # Channel Presentation Observations: Braintrust ## Source Scope - **Video ID:** 01a15ca6-e9d4-7ca5-af2a-a3025bb251b4 - **Title:** "Why Building an Eval Platform Is Harder Than It Looks — Braintrust" - **Format:** Recorded conference-style session. The speaker opens with "Thank you for joining this session" and refers to an earlier session by "Jess." - **Metadata:** None supplied beyond the title. No timestamps appear in the transcript, so none are cited. --- ## 1. Explicit Self-Description (Creator Claims) ### 1.1 Speaker identity - **Name and role:** The speaker says, "My name's Hussein. I lead the solution engineering organization on the West for Braintrust." - **Background:** "I spent 15 years in solutions between Salesforce and Databricks. I've been a lifelong technologist. I've built computers since I was 5 years old." - **Note:** These are personal claims by the speaker, not verified facts about the channel. ### 1.2 Company self-description - **Core focus:** "Braintrust as a platform is focused around agent quality. And the core idea is helping teams build and maintain confidence with the AI features or the AI agents that they're shipping." - **Two pillars named:** - **Evals:** "what you do before your agent reaches production." - **Observability:** "once your agent is in production and is interfacing with real-life users." ### 1.3 Self-described product scope - The speaker describes Braintrust's own platform internals: "we built an abstraction layer called BTQL," and "we log your tracing data into our platform, we can run inference on them." - The speaker claims Braintrust can "surface these insights automatically" and identify "unknown unknowns." --- ## 2. Observable Presentation Patterns ### 2.1 Intended audience The transcript demonstrates an audience of technical and cross-functional practitioners: - **AI and product engineers:** "a product engineer... they see this problem and they think, 'Okay, I can build a nice UI.'" - **Product managers:** "There are PMs who used to build PRDs that are now running evals." - **Subject-matter experts:** "There are SMEs who have the domain expertise of the product that you're building." - **Conference attendees:** The speaker polls the room repeatedly ("How many of you know what Braintrust is and what we do as a company? Just show of hands."). ### 2.2 Claimed scope of discussion The talk claims to cover: - Why evals matter, framed as managing non-determinism: "LLMs by nature are non-deterministic. They are highly variable." - A maturity progression for eval tooling: 1. Spreadsheets 2. Bespoke UIs 3. Side-by-side experimentation 4. Production-linked flywheels 5. Coding-agent-driven evals - The systems engineering behind eval platforms: "building an eval platform truly becomes a systems problem." ### 2.3 Narrative structure - **Problem framing first:** The speaker defines evals and explains the risks of skipping them (brand, compliance, cost and maintenance). - **Progressive maturity model:** Each stage is introduced with its limitations, e.g., "the returns diminish pretty quickly," and "collaboration is very weak." - **Product connection:** Platform capabilities are introduced as the answer to stated gaps. ### 2.4 Tone and interaction style - **Conversational and audience-directed:** "So, there's no shame in that, by the way." - **Reassuring toward beginners:** "spreadsheet-based Evals are not wrong. However, they're often the first step." - **Self-deprecating or deflecting about the product:** "I won't spend much time talking about Brain Trust," followed by a substantial technical description of its ingestion and query needs. --- ## 3. Product Promotion Evidence The transcript contains product-forward language alongside educational content: > "BrainTrust can surface these insights automatically. Because we log your tracing data into our platform, we can run inference on them and we can tell you the unknown unknowns." > "you can tell your coding agent of choice, 'Find me all traces in the last 24 hours where the user had a poor experience. Run the evals for me.'" - **Observable pattern:** Educational framing is paired with product capability claims. The transcript does not present competitor comparisons or pricing. --- ## 4. Claims Versus Demonstrated Content | Category | Creator Claim | Demonstrated in This Video | |---|---|---| | Agent quality focus | "Braintrust... is focused around agent quality" | Consistent framing of evals and observability as one loop | | Scale handling | Traces are "hundreds of megabytes" and require specialized querying | Described conceptually; no live demo or data shown | | Automated insight | Surfaces "unknown unknowns" automatically | Asserted only; no example output shown | | Coding-agent integration | Coding agents can run evals and log results | Described via a single example prompt; no demonstration | --- ## 5. Limitations of This Observation - The transcript is a single session. It cannot establish channel-wide patterns, publishing cadence, or other content. - No curated metadata was supplied, so channel-level identity beyond this session is not assessed. - All product claims are the speaker's assertions and are unverified by this material. # Delivery Observation: Why Building an Eval Platform Is Harder Than It Looks — Braintrust ## Overall Tone - **Conversational, low-pressure presentation style.** Filler words ("um," "you know," "like") appear throughout, e.g., "you know, evals matter because LLMs by nature are non-deterministic." - **Reassuring and non-judgmental toward the audience.** He repeatedly normalizes spreadsheet-based evals: "So, there's no shame in that, by the way. It's actually the a good first step," and later "There's nothing to be shameful about." ## Audience Interaction Habits (Recurring) - Uses **show-of-hands polls** as a recurring device: > "How many of you know what Braintrust is and what we do as a company? Just show of hands." > "Just show of hands, how many of you have run Evals and logged the results into a spreadsheet?" - Acknowledges prior context: "I know some of you were in the room earlier when Jess did her session." - Offers post-talk availability: "anything I don't get to, you feel free to come up afterwards and we can chat about." ## Rhetorical & Structural Patterns - **Anticipates audience objections rhetorically:** > "So, you might be thinking, 'Okay, well, Evals is just a UI on a spreadsheet, right?'" - Uses a **reflexive hypothetical question** late in the talk: "And then it becomes a question of like, so so what? Why is this a Why is this important?" - **Staged/progressive structure**: phases of eval maturity (spreadsheet → bespoke UI → side-by-side experimentation → production flywheel), each with advantages then limitations ("But the limitation that you might have thought about, it continues"). - **Recurring metaphor vocabulary:** - Iceberg: "there's a lot more that happens behind the scenes or underneath the iceberg." - Hill climbing: "Evals are a way for you to continue hill climbing against one target, which is agent quality." - Flywheel / improvement loop: "They want to build the improvement loop" and later "You get teams that are building the flywheel." - Uses **contrastive framing** ("powerful vs. risk," "offline vs. production"): "that variability is exactly what makes them so powerful... But that same flexibility also creates risk." ## Recurring Phrases & Verbal Tics - **"You know"** appears many times as a connective. - **"kind of"** frequently: "which finds me kind of on the bleeding edge," "get into the loop of doing evals," "when we think kind of going back to the flywheel." - **Immediate word repetition** under delivery pressure: "the the returns diminish pretty quickly," "The big biggest advantage," "you have to have you have to maintain it," "snapshots of of of in-time data," "my my agent experience." - Self-correcting false starts: "This is where things start to get complicated." ## Teaching Habits - **Defines terms plainly before using them**: "The first is evals... The second is observability." - **Numbered/three-part breakdowns**: "It has a way to execute agents against test inputs. Second, a way to view the outputs or the scores... And third, a set of test inputs." - **Concrete illustrative examples**: "logging a movie's worth of data in each interaction, hundreds of megabytes," and a sample natural-language agent command: "Find me all traces in the last 24 hours where the user had a poor experience. Run the evals for me." - **Transitions that manage time explicitly**: "Now, I won't spend much more time here," "I won't spend much time talking about Brain Trust." ## Vocabulary Style - Technical/product vocabulary: "non-deterministic," "failure modes," "drift," "semi-structured JSON," "POC," "regressions," "observability." - Sales-adjacent framing near the end ("we can tell you the unknown unknowns") — the talk shifts from neutral teaching to vendor pitch in the final third, observable in the shift to "we built" and "we can surface these insights automatically." ## Stability vs. Isolated Moments - **Stable patterns** (multiple instances): show-of-hands polling, "you know"/"kind of" fillers, immediate word repetition, staged maturity narrative, reassurance about spreadsheets, contrastive advantage/limitation framing. - **Isolated moments**: the personal credential opening ("I've built computers since I was 5 years old"), the iceberg metaphor, and the "movie's worth of data" comparison appear once each — vivid but not repeated habits. ## Uncertainty Notes - Pauses and overlapping timestamps suggest a live talk; disfluencies may be underrepresented or overrepresented in the transcript, so frequency claims for fillers are approximate. # Observation: "Why Building an Eval Platform Is Harder Than It Looks" (Braintrust) ## Source Overview - **Video ID:** 01a15ca6-e9d4-7ca5-af2a-a3025bb251b4 - **Speaker:** Hussein, who introduces himself in the transcript as leading the solution engineering organization for the West at Braintrust. He says he spent 15 years in solutions roles between Salesforce and Databricks. - **Format:** Conference session talk. The speaker refers to an earlier session by "Jess" and to audience hands raised in response to his questions. - **Source basis:** Transcript and title only. No other curated speaker metadata was supplied. --- ## 1. Core Framing: Agent Quality The speaker presents **agent quality** as the central concern. He describes Braintrust as a platform "focused around agent quality," aimed at helping teams "build and maintain confidence" in AI features or agents they ship. He names two pillars: - **Evals** - Run *before* the agent reaches production. - Teams "experiment, they test the behavior, and they build the confidence" needed before shipping. - **Observability** - Runs *after* the agent is in production, interacting with real users and "creating interactions." - Purpose: take the hypothesis built offline and "reinforce that in your production scenario through continuous monitoring." > "Evals and observability are closely related to the same problem. One happens in production, the other one happens in development, and both is the understanding to improve agent quality." --- ## 2. Why Evals Matter (Stated Reasoning) ### Premise LLMs are **non-deterministic** and **highly variable**. The speaker frames this variability as both the source of their power and a source of risk. ### Argument - Agents use an LLM as their "brain," and agentic experiences are "starting to become the primary way" users will interact with companies. - Teams therefore need confidence that agents will "behave reliably and produce the outcomes that they expect." ### Risks named without evals - **Brand:** inconsistent behavior. - **Compliance:** the agent "says or does the wrong thing." - **Cost and maintenance:** systems that are "hard or unreliable to debug or control." ### Stated goal Reduce uncertainty before launch so customers have a good experience and agents behave as expected. --- ## 3. The Spreadsheet Starting Point The speaker poses the view that "Evals is just a UI on a spreadsheet." He then responds to it through audience polling, noting that spreadsheet-based evals are a legitimate starting point. ### Simplest eval system (three criteria) 1. A way to **execute agents** against test inputs. 2. A way to **view outputs or scores**, even if logged in a spreadsheet. 3. A **set of test inputs** that can invoke the agent (a prompt, request, or context). ### Qualifications - The speaker says spreadsheet evals are "not wrong" and "often the first step." - He also says: "there's more to the iceberg than just what we just talked about." --- ## 4. Maturity Stages (As Described by the Speaker) The speaker describes a progression he says is commonly observed. Each stage is paired with stated limitations. ### Stage 1: Spreadsheets - **Advantage:** accessibility and speed, with "basically zero barrier to entry." - **Limitations:** - "Returns diminish pretty quickly." - Mostly documentation; "You can't really do true experimentation." - Analytics are hard and manual. - Human scoring is valuable but difficult. - Collaboration is weak. - Domain experts are "usually boxed out." ### Stage 2: Bespoke UI - Usually built by a product engineer. - **Gain:** brings in personas such as PMs, who can now participate in evals. - **Persisting limitations:** - Documentation and review remain difficult to collaborate on. - Iteration is faster, but persisting data for "long horizon analytics" is still hard. - The speaker calls it "still just a reporting tool." ### Stage 3: Interactive Experimentation - Non-technical users can experiment, not just log results. Examples given: tweaking a system prompt, changing the underlying model, iterating on parameter values. - Enables side-by-side comparison. - **Remaining gap (stated):** test cases are still "a curated set" from offline evals. Once shipped, "I'm operating in the blind." ### Stage 4: Closed Improvement Loop (the "North Star") - Observe failure modes in production using defined dimensions. - Log every input, output, and step of the trace. - Convert failure modes into offline test cases. - Iterate offline without introducing new regressions. - Ship and continue "hill climbing." - The speaker notes that **drift** is "a real thing" because user interactions and agent-building practices both change over time. --- ## 5. The Improvement Loop as a Systems Problem ### Claimed difficulty Once teams build the flywheel, they "have to maintain it" and "own this product that you've built." The speaker says this is manageable at POC or demo scale but harder in production. ### Data characteristics (as stated) - Agent traces are "semi-structured JSON." - Some teams log "a movie's worth of data" per interaction, reaching "hundreds of megabytes." - Traditional heartbeat observability data is described as "just kilobytes." - The speaker claims typical cloud data warehouses "kind of collapse under this type of volume and load." > "Building an eval platform truly becomes a systems problem." ### Two workloads (as stated) - **Real-time ingest and query:** to learn in real time why users are having poor experiences. - **Long-running queries:** for fine-tuning and for humans annotating judge outcomes to achieve alignment. ### Other system concerns mentioned - An abstraction layer the speaker calls **BTQL**, offered as a query interface for users. - Underlying database design. - Access controls, permissions, and data masking. - Aggregation over large volumes as well as point-in-time snapshots. The speaker says the system should empower not only AI engineers but also PMs and SMEs. --- ## 6. Agents as Eval Operators The speaker says agents have become "a first-class citizen" of eval platforms. - Users can issue natural-language requests in a headless setting. His example: "Find me all traces in the last 24 hours where the user had a poor experience. Run the evals for me." - Coding agents with access to infrastructure and code become "the mechanism to run the evals and log those results." --- ## 7. Automated Insight and Human Review ### Stated Braintrust approach - Rather than humans defining every success and failure dimension, Braintrust "can surface these insights automatically" by running inference on logged tracing data. - Examples of "unknown unknowns" the speaker gives: - Scenarios where the agent is "failing silently." - Users showing frustration by repeating the same request. ### Human role (as stated) - Coding agents can iterate on changes and suggest improvements. - "Ultimately, the humans responsibility is to review the outcome." Humans choose which iteration to ship. --- ## 8. Tensions, Qualifications, and Open Items | Topic | Stated position | Qualification or tension | |---|---|---| | Spreadsheets | "Not wrong" and "a good first step" | Also described as hitting diminishing returns and being hard to evolve | | Human involvement | Humans must be "deeply involved" in the manual process, which "could be laborious" | Automation is presented as reducing this, yet the speaker also says humans must still review outcomes | | Data volume | Warehouses "kind of collapse" under trace volume | Stated qualitatively. No benchmarks, sources, or named comparisons given | | Brand/product claims | Braintrust can surface insights automatically and handle real-time ingest | These are the speaker's descriptions of his company's product. No independent evidence presented | | Coding agents | Presented as a mechanism to run and log evals | The speaker does not discuss failure modes or reliability of such agents | | Scope | "There's a lot more that happens behind the scenes" | Topics named but not covered: underlying database, permissions, data masking, scoring systems, review processes. The speaker deferred these to follow-up conversation | ### Explicitly unresolved - The speaker says he will not cover everything and invites follow-up discussion. - No specific numbers, customer examples, or comparative studies are cited. --- ## 9. Summary of Key Claims (Attributed to Hussein) - **Agent quality** depends on pre-production **evals** and post-production **observability**, which should form a single loop. - **LLM non-determinism** makes evals necessary to reduce brand, compliance, and maintenance risk. - Eval maturity typically moves from **spreadsheets → bespoke UI → interactive experimentation → closed production-to-offline loop**. - Production **trace data** at scale is a **systems problem**, not only a UI problem. - **Agents** are increasingly used to run and log evals through natural language. - Humans remain responsible for **reviewing outcomes** and choosing what to ship. # Speaker Attribution and Provenance: "Why Building an Eval Platform Is Harder Than It Looks — Braintrust" ## Video Identifier - **Video ID:** 01a15ca6-e9d4-7ca5-af2a-a3025bb251b4 - **Title (as supplied):** Why Building an Eval Platform Is Harder Than It Looks — Braintrust - **Timestamps:** None supplied in the transcript. ## Speakers ### Primary Speaker: "Hussein" - **Name:** Hussein (first name only, as stated in the transcript; no surname given) - **Stated role:** Leads the solution engineering organization for the West at Braintrust. - **Stated background (self-reported):** 15 years in solutions roles between Salesforce and Databricks. - **Stated affiliation:** Braintrust. - **Speaking context:** Presents a session at an event; refers to an earlier session by "Jess," who is not a speaker in this recording. - **Speaking span:** The entire transcript. No other speaker is identified in the transcript. ### Other Individuals Referenced (Not Speakers Here) - **Jess:** Named as having presented an earlier session. No content from Jess appears in this transcript. - **Audience members:** Responded by show of hands to questions about familiarity with Braintrust and with spreadsheet-based evals. Their responses are not transcribed beyond the speaker's summaries ("I see one or two"). ## Promotional Content (Flagged) > **Promotional segment flag:** The speaker is employed by Braintrust, and the presentation promotes Braintrust's platform. Promotional content is identified below. - **Self-introduction and company framing:** The speaker introduces his role and asks the audience how many know "what Braintrust is and what we do as a company." - **Product positioning:** Braintrust is described as "focused around agent quality," with two named pillars, evals and observability. - **Explicit product references:** - The speaker says "I won't spend much time talking about Brain Trust" while describing its ingest and query architecture. - The speaker describes **BTQL** as "an abstraction layer" Braintrust built for querying trace data. - The speaker states that "BrainTrust can surface these insights automatically," including by running inference on logged tracing data. - Platform features mentioned include the underlying database, permissions and access controls, and data masking. - **Calls to action:** The speaker invites audience members to "come up afterwards" to chat. ## Provenance Inventory ### Referenced External Sources - **None identified.** The transcript names no papers, publications, URLs, datasets, or third-party reports. ### Third-Party Claims - **None explicitly attributed to outside parties.** General statements about the field (for example, that LLMs are non-deterministic, that agentic experiences are becoming a primary interaction mode, and that "drift becomes a real thing") are presented in the speaker's own voice without a cited source. ### Unnamed Organizations and Teams (Speaker's Observations) - The speaker refers to "some teams" and "many teams" without naming them. Examples include teams logging large trace payloads and teams that have built internal flywheel workflows. - The speaker describes a problem as one "we ran into," which is an unquantified claim about Braintrust's own experience. ### Attributed Historical Context - The speaker refers to "the chat GPT boom" a few years ago as the point when real-time ingest of large payloads became a requirement. This is a general reference with no cited source. ### Speaker Assertions Without Cited Evidence - Claims about trace payload sizes ("a movie's worth of data," "hundreds of megabytes") and comparisons to traditional observability ("kilobytes in size") are stated without a cited source. ## Availability Limits | Channel | Status | |---|---| | Transcript | Available, machine-style, with artifacts (`\>\>`, `\>`, `[music]`) and no speaker labels beyond the opening | | Timestamps | Not supplied | | Curated metadata beyond title | Not supplied (no description, speaker list, chapter markers, or sponsor disclosure) | | Comments | Availability not indicated | | Slides or visuals | Not supplied; the speaker refers to a "spreadsheet" and "UI" without visual confirmation | | Earlier session by Jess | Not supplied | ## Attribution Notes for Later Authors - Any statement in this transcript should be attributed to **Hussein (Braintrust)** unless the transcript explicitly says otherwise. - Product claims about Braintrust come from a company employee in a promotional presentation. - No external sources were cited, so any claim requiring independent support must be sourced outside this transcript. \>\> Thank you for joining this session on why building agent quality platforms is hard. My name's Hussein. I lead the solution engineering organization on the West for Braintrust. Um, I spent 15 years in solutions between Salesforce and Databricks. I've been a lifelong technologist. I've built computers since I was 5 years old. So, I have this like insatiable appetite for technology, which finds me kind of on the bleeding edge of AI here at Braintrust. For those of you, I know some of you were in the room earlier when Jess did her session. I saw some hands about evals. How many of you know what Braintrust is and what we do as a company? Just show of hands. Perfect. I see one or two. So, at a glance, Braintrust as a platform is focused around agent quality. And the core idea is helping teams build and maintain confidence with the AI features or the AI agents that they're shipping. And there are two key pillars of agent quality. The first is evals, which we talked about earlier. Evals is what you do before your agent reaches production. This is when teams experiment, they test the behavior, and they build the confidence that they want to have before they ship to production. The second pillar is observability. Observability is once your agent is in production and is interfacing with real-life users. It's creating interactions. And the goal is to take that hypothesis that you built in your offline development process and then reinforce that in your production scenario through continuous monitoring. Evals and observability are closely related to the same problem. One happens in production, the other one happens in development, and both is the understanding to improve agent quality. Now, I won't spend much more time here. Let me close that. Let's talk about why we're here, which is why are evals important? You know, evals matter because LLMs by nature are non-deterministic. They are highly variable. That variability is exactly what makes them so powerful. They can reason across many different domains. They can solve different kinds of problems, and they can handle a wide range of user needs. But that same flexibility also creates risk. Agents use LLM as their brain. And agentic experiences are starting to become the primary way in which users will interact with companies. So teams need confidence in their agents that they will behave reliably and produce the outcomes that they expect. And without evals, companies face real risks. Brand Brands become a risk if there's inconsistent behaviors. Compliance if the agent says or does the wrong thing. And there's a cost and maintenance risk if the systems are hard or unreliable to debug or control. The goal of evals is to reduce that uncertainty before launch so that customers have a good experience and the agents behave the way that you expect them to. So you might be thinking, "Okay, well, Evals is just a UI on a spreadsheet, right?" And I guess I should ask the audience. Just show of hands, how many of you have run Evals and logged the results into a spreadsheet? Has that happened? Okay, I see one or two. So, there's no shame in that, by the way. It's actually the a good first step. The important thing is that teams acknowledge that this is a real problem and they need a way to understand how agents behave based on different inputs. At the simplest level, an eval system tests against or it has three different kind of criteria. It has a way to execute agents against test inputs. Second, a way to view the outputs or the scores, even if those are being logged in a spreadsheet. And third, a set of test inputs or examples that can invoke the agent. An input example is whatever information is needed to start the agent run. It could be a prompt, a request, or context that causes the agent to act. Now, spreadsheet-based Evals are not wrong. However, they're often the first step for a useful version of eval workflow. But, of course, as you can imagine, there's more to the iceberg than just what we just talked about. And if Evals were only, you know, run the agent, view the output, and then score it in a spreadsheet, this talk would be over right now. But, there's a lot more that happens behind the scenes or underneath the iceberg. There are many supporting teams that eventually need to build better data sets, scoring systems, review process, debugging tools, and the way to connect pre-production testing with production behavior. We'll touch on some of these today, and anything I don't get to, you feel free to come up afterwards and we can chat about. And this is where things start to get complicated. The underlying technology is complex. LLMs are not simply deterministic. Agent quality is a multi-persona problem. It's not just software engineers and AI engineers. There are PMs who used to build PRDs that are now running evals. There are SMEs who have the domain expertise of the product that you're building. And evals themselves are just one part of the development and operating workflow. And gone are the days where you build once with unit tests and regression tests, you ship to production, and you don't think about it. Evals are a way for you to continue hill climbing against one target, which is agent quality. So, let's talk about the different stages of the the eval platforms that we see. And before we do that, this is the North Star for many teams. They want to build the improvement loop, which is I have a feature or application that's in production. I can observe the failure modes that happen based on dimensions that I define. I want to be able to grab those failure modes and create test cases so that I can iterate offline, so I can improve my agent quality without introducing new regressions. And then you continue iterating over time, because just like in classical ML, drift becomes a real thing. The way users interact with your agents will change over time. The way you build your agents will change over time. So, what What phase one look like? There were some hands that were raised around spreadsheets and it is not There's nothing to be shameful about. This is where many people start. And as the basic setup is pretty simple. It's a four loop with a set of input examples and a way to execute the agent. And with that, you can run the same examples that you tweak and see how the agent responds with the outputs over time. The big biggest advantage is accessibility and time. You can start this with basically zero barrier to entry. But the the returns diminish pretty quickly. At this stage, you're pretty much only documenting. You can't really do true experimentation. And let alone do any kind of type of analysis over time. Common limitations are analytics are hard, they're manual, human scoring is valuable but difficult to do, collaboration is very weak, and domain experts are usually boxed out of these type of results. So, again, spreadsheets are a great starting place but difficult to evolve. Then someone, usually a product engineer, you know, they see this problem and they think, "Okay, I can build a nice UI, a bespoke UI to help serve to solve this problem." And while it might be nice at a glance, um it helps you bring in the different personas that you would otherwise not have access to. This allows PMs to now kind of get into the loop of doing evals. But the limitation that you might have thought about, it continues. That the documentation, the review process, though it might visually look nicer, still collaboration becomes difficult. And though iteration cycles become quicker, it is still not easy for users to persist that data and do long horizon analytics on. Still just a reporting tool. Then the next evolution becomes hey, I want my non-technical users to have an interface where not that they just log the results, but they should be able to experiment. I want to be able to tweak a system prompt. I want to change the underlying model. I want to iterate on some parameter values. So, being able to do this type of side-by-side comparison becomes the next evolution of the eval cycle that we see. But, there's still one primary gap. And that is that the test cases in which you use to do these kind of comparisons are still a curated set that arise in your offline evals. But, what happens in production? I've done my iterations here. I have a good idea of what I can expect to happen in prod, but once I ship it to production, I'm operating in the blind. So, I have no idea what steps my agent is taking and when those failures are actually happening in my live environment. And when we think kind of going back to the flywheel, when we think about what are the best systems that enable this, well, you should be able to starting at the top observe the failure modes, log every input, output, every step of your trace execution that your agent takes, analyze, understand what went wrong. You create the dimensions of failure and success because you know the outcomes. Then grabbing those failure modes, those different scenarios where your users had poor interactions, and creating evals in them in your offline process, iterating on them so you can create the improvements without developing new regressions. And then shipping to production and hill climbing against this. And when you think about the final outcome of this, well, you get teams that are building the flywheel. You get that entire development cycle inside of your platform, but then the problem becomes you have to have you have to maintain it. You own this this product that you've built. And especially this is easy to do when it comes to smaller scale or POC and demo environments. But agent traces are nasty. They are semi-structured JSON. In some cases we see teams that are logging a movie's worth of data in each interaction, hundreds of megabytes. So being able to query that type of data can be difficult. And your typical cloud data warehouses kind of collapse under this type of volume and load. And this is the same problem that we ran into. I won't spend much time talking about Brain Trust, but when you have production traces coming in, you want to be able to query that data in real time as it lands in there. Because if your users are having poor experiences, you want to be able to know in real time what is happening, why are they having these poor experiences, and how can I remedy that as soon as possible. You also want to be able to do long-running queries. If you think about being able to fine-tune your your agent experience or even have humans who are doing alignment by annotating your your judges' outcomes. Two different workloads. So we built an abstraction layer called BTQL, which is a another form of complexity because you want an interface for people to be able to query that data. So, it's not necessarily that it's just a UI or UX problem, but building an eval platform truly becomes a systems problem. And there are novel set of issues when the chat GPT boom happened a few years ago where I need to do real-time ingest. I have huge payloads that are tens of megabytes, hundreds of megabytes in size sometimes, compared to traditional heartbeat observability, which are just kilobytes in size. The structure and the shape of the data is different. Having deeply nested semi-structured full text is difficult to query, and the read patterns are different as well. You want to be able to aggregate in large volumes, but also be able to do snapshots of of of in-time data. And building the right system should allow you to not only empower the AI engineer, but also the PMs, as well as the SMEs. And more recently, we've seen that agents have become a first-class citizen of these eval platforms where you want to be able to use natural language in a headless experience where you can tell your coding agent of choice, "Find me all traces in the last 24 hours where the user had a poor experience. Run the evals for me." And because these coding agents have access to all of your underlying infrastructure and your code base, they become the mechanism to run the evals and log those results. And then it becomes a question of like, so so what? Why is this a Why is this important? Well, the goal is everything that I've shown you so far, it requires the humans, the the engineer, the PM to be deeply involved in this process and it could be laborious at times. I BrainTrust, the way we think about this is we want to help you operate at scale. Rather than you having to think about these dimensions and of success and failure for your AI agent, BrainTrust can surface these insights automatically. Because we log your tracing data into our platform, we can run inference on them and we can tell you the unknown unknowns. Hey, what are what are scenarios and when my agent is failing silently or what are scenarios that my users are experiencing frustration by prompting my agent for the same request over and over again. And it's a lot of things that we didn't talk about. Things like the underlying database that powers it or having to build our back into the system to manage controls and permissions or data masking. You know, each one of these features proves that this is not just a UI and a spreadsheet, but rather it is a a systems problem that powers it. And as we think about the evolution of of the improvement loop, I said this earlier where humans were doing the improvement loop and now we see coding agents where they could iteratively make changes and suggest what kind of improvements you should be making to your application. And ultimately, the humans responsibility is to review the outcome. If I have different iterations of these e-vals that are run by my coding agent, I can look at the outcome and make the decision of which version of it is the best for what I want to ship to production. So, with that, that is the end of my talk. I appreciate you all coming out and learning about why it's hard to build eval systems. Thank you. \>\> \[music\]