Marina Petzel (GenAI Advocate, Datadog) explains why the classic golden signals (latency, errors, traffic, saturation) are necessary but insufficient for generative AI applications, and which metrics to add on top. Topics: cost monitoring (token creep, model drift, uncached calls, tagging for cost attribution), safety monitoring (prompt injection, PII leakage, toxicity, jailbreaks), and quality monitoring (hallucination rate, relevance, user satisfaction, completeness, RAG retrieval quality). Recorded at the AI Engineer World's Fair 2026 in San Francisco. # Video Identity Observations ## Source Overview - **Video:** "Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog" - **Format evidence:** A conference talk, given the direct audience address (*"Hello everyone. Thank you for joining me today"*) and references to a *"booth here"* and meeting attendees afterwards. ## Self-Description Evidence (Explicit Claims) ### Speaker identity - The speaker states: *"my name is Marina Petzel. I am gen AI advocate here in DataDog."* - The title confirms the Datadog affiliation: *"Marina Petzel, Datadog."* ### Organizational positioning - The speaker describes Datadog's market position: *"Datadog has your back here since we are working in observability and monitoring space for for really long time."* - A product claim is made: *"we do have our agent observability uh product which allows you to quickly and easy understand if your AI application is healthy."* ### Commercial intent (stated) - The speaker invites follow-up: *"please feel free to see me afterwards. We also do have the booth here,"* and mentions *"QR codes for the free trials"* and *"a really good um offers."* This is a vendor talk with explicit product promotion, not neutral technical content. ## Intended Audience (Observable) - **Engineers building generative AI applications.** The talk assumes production experience: *"for us as an engineer is becoming more and more critical right now to track it all."* - **Teams operating LLM apps in production**, evidenced by operational framing: *"you must constantly evaluate quality in the live environment"* and concerns about *"spending out of control."* - **Practitioners familiar with traditional observability.** The talk opens from shared knowledge: *"for years, many of us monitored our applications with the golden signals"* (latency, error, traffic, saturation). Attack vectors are also assumed known: *"We used to worry about SQL injection or cross-site scripting before."* ## Claimed Scope - The talk claims to extend traditional monitoring, not replace it: *"We are not saying that we need to get rid of traditional understanding… but also on top of that, we need to put another layer."* - The claimed scope covers three GenAI-specific layers: **cost, safety, and quality** monitoring, with concrete metrics (token creep, prompt injection rate, hallucination rate, RAG quality, etc.). - A software-development framing is established as the thesis: *"Traditional metrics will tell you if your system is up and running, but it won't tell you how well it's running."* ## Demonstrated Presentation Patterns (Not Claims) - **Metric-and-threshold orientation:** The talk consistently pairs metrics with target numbers (e.g., PII *"0% of your production outputs,"* user satisfaction *"more than 80 85% of the positive feedback"*), demonstrating a prescriptive, engineering-practice style. - **Traditional→GenAI contrast structure:** Nearly every section contrasts predictable traditional systems with nondeterministic GenAI behavior. - **Vendor-demo softness:** Product specifics are deferred to in-person contact (*"if you want to hear more about that, please feel free to see me afterwards"*) rather than demonstrated on stage. ## Synthesis Notes for Channel-Level Analysis - This single video suggests a **vendor-expert educational persona**: technical conference content from a Datadog employee that is genuinely substantive but culminates in product promotion. - Confidence note: observations here derive from one conference talk; channel-wide patterns cannot be confirmed from this material alone. # Delivery Observation — Marina Petzel, "Your LLM App Returned 200 OK. It Was Still Wrong." ## Tone & Register - **Conversational-lecture tone** with frequent direct audience address: *"I'm sure many of you got some response from modern AI applications, right?"* - **Rhetorical questions as section openers** are a consistent habit: *"And what are the problems you might encounter with the cost?"*, *"How we can track the spending before it uh spirals out of control?"* - **Shared-experience framing** to build rapport: *"we all know that the same prompt can generate different responses every time we run it"*, *"We used to worry about SQL injection or cross-site scripting before."* - Occasional light humor anchored in shared pain: *"your AI can be cheap, but still leak some of the sensitive or personal information, which is not great."* ## Recurring Phrases & Verbal Signatures - **Tag question "right?"** appears repeatedly as a comprehension check: *"the same input always yield the same output, right?"*, *"it's predictable, right?"*, *"our journey application is healthy, right?"* - **"200 OK" as a recurring motif** — appears in the title, mid-talk, and again near the end: *"200 okay response from your server doesn't necessarily mean that the answer was helpful"*, *"200 okay response from your server sometimes can be a lie if the answer is hallucination."* This is the talk's central framing device, deliberately restated. - **"Now let's imagine that…"** as a transition device between sections: *"let's let's imagine that we handle all the cost issues"*, *"Now, let's imagine that cost is checked, security is checked…"* This creates a cumulative "checklist" narrative arc (cost → safety → quality). - **"And now let's talk about…" / "And finally…"** signal every section boundary. - **Evaluative intensifiers** recur: *"very crucial problem"*, *"arguably the most crucial dimension"*, *"essential"* appears many times (*"it's become essential"*, *"essential to track"*). ## Teaching Habits - **Numbered-list scaffolding throughout.** Nearly every section is delivered as an enumerated set: - Three cost problems: *"the first is a token creep… Second is the model drift… And finally, it's on hash code."* - Four tagging levels: *"first layer what you can implement is the feature level… The second level is the use user level… Third is the model level… And the finally, endpoint level tagging."* - Five quality metrics: *"We recommend to focus on these five metrics. One of them is hallucination rate…"* - **Concrete quantification as a persuasion habit.** Each problem comes with a striking number: *"might result up to eight x cost increased"*, *"can be higher 15 times higher uh per the cost request"*, *"70% of the spend sometimes are redundant"*, *"only acceptable target should be 0%"*, *"more than 80 85% of the positive feedback."* - **Problem → mechanism pairing.** After naming each risk, she immediately names a technique: PII → *"regex or named entity recognition"*; relevance → *"similarity using bird or embedding similarity"*; completeness → *"LLM as a judge."* - **Comparative old-world vs. new-world framing** is the talk's backbone: *"Traditional compute cost are predictable… But for gen AI, this this is dynamic variable here."* ## Communication Patterns & Fluency Notes - **Frequent false starts and self-repairs**, suggesting live unscripted delivery: *"uh and and a month it might cost you"*, *"this this is dynamic variable here"*, *"let's let's imagine"*, *"how we can determine if your content is…"* - **Non-standard article and verb-agreement usage** ("a simple 500 error code can will never catch", "the same input always yield"), indicating English as an additional language; meaning stays recoverable throughout. - **Consistent signposting discipline** despite the disfluencies — the structural skeleton (numbered points, "and finally", "let's summarize") is always audible, which keeps the talk navigable. ## Stable Patterns vs. Isolated Moments - **Stable:** rhetorical questions, "right?" tags, enumerated structures, quantified claims, cost/safety/quality sequencing, and the "200 OK" motif — each recurs across multiple sections. - **Thin evidence (single occurrence):** the humor ("budget hawks", "which is not great") and the product pitch closing (*"Datadog has your back here"*) each appear only once, so no repeated persuasion or selling pattern can be confirmed from this transcript alone. # Observation Record: "Your LLM App Returned 200 OK. It Was Still Wrong." **Speaker:** Marina Petzel, identified in the transcript as a "gen AI advocate here in DataDog" **Video:** Conference talk on monitoring metrics for generative AI applications. --- ## Central Thesis Petzel argues that the traditional **"golden signals"** — latency, errors, traffic, saturation (LETS) — remain necessary but are **no longer sufficient** for healthy gen AI applications. As stated: > "Traditional metrics will tell you if your system is up and running, but it won't tell you how well it's running." She frames her talk around the claim that a **200 OK response does not guarantee correctness**: "200 okay response from your server sometimes can be a lie if the answer is hallucination." --- ## Four Differences Between Traditional and Gen AI Applications (per Petzel) 1. **Non-determinism** — "The same prompt can generate different responses every time we run it," so standard regression testing alone is insufficient; quality must be evaluated continuously in live environments. 2. **Variable cost structure** — Costs vary by token count, model choice, and context window size, requiring real-time granular cost tracking. 3. **New attack vectors** — Prompt injection, jailbreaking, and PII leakage in outputs replace (or supplement) SQL injection and XSS. She notes a "simple 500 error code will never catch" these security issues. 4. **Subjective quality** — Output is judged on a spectrum (relevant? accurate? complete?) rather than binary working/not-working. --- ## Position 1: Cost Monitoring ### Three cost failure modes identified - **Token creep** — Quietly expanding context windows (e.g., from "4,000 tokens" to "132" — likely "132K") without financial review; she claims context growth "might result up to eight x cost increased." - **Model drift** — Switching to more powerful models (her example: "Hi Cool 4.5" to "Opus 4.8," plausibly Claude Sonnet/Opus); the pricier model "can be higher 15 times higher per the cost request." - **Hash cache absence (no caching layer)** — Repeated identical queries hit the API; citing "our own research," she claims **"70% of the spend sometimes are redundant."** ### Cost attribution strategy: "You must tag absolutely everything" | Level | Tag examples | Stated purpose | |---|---|---| | **Feature** | chat vs. summarization | Attribute cost to product areas; inform roadmap ("budget hawks") | | **User** | user ID, org name | Chargebacks, billing, detecting abusive usage | | **Model** | model name, provider (OpenAI, "Tropic") | Benchmark model-vs-model expense | | **Endpoint** | region, production vs. staging | Infrastructure planning | --- ## Position 2: Safety Monitoring Four safety metrics, with her stated targets: - **Prompt injection rate** — Detected via pattern matching or classifier models; "target for very aggressive thresholds." - **PII detection rate** — Scan outputs for SSNs, credit cards, personal data via regex or NER; **"only acceptable target should be 0%"** in production. - **Content moderation score** — Toxicity classifiers to score risk of toxic/harmful/biased output; minimize as much as possible. - **Jailbreak attempts** — Track patterns indicating system prompt overrides; "ideally... block all the 100% of the attempts." She acknowledges cost control alone is inadequate: "Your AI can be cheap, but still leak some of the sensitive or personal information." --- ## Position 3: Quality Monitoring (called "arguably the most crucial dimension") Five recommended metrics: 1. **Hallucination rate** — Percentage of responses containing claims unsupported by grounded data; combine manual review with automated fact-checking; "aggressive targets." 2. **Relevance scores** — BERT or embedding similarity to score whether the response addresses the question. 3. **User satisfaction** — Thumbs up/down, star ratings, NPS; stated target: **more than 80–85% positive feedback**. 4. **Answer completeness** — Evaluated via **LLM-as-a-judge** for whether the response holistically addressed user intent. 5. **RAG quality** — If using retrieval-augmented generation, track whether fetched documents were relevant (e.g., top-K accuracy, normalized discounted cumulative gain). --- ## Summary Position and Qualifications - **Explicit qualification preserved:** She states, "We are not saying that we need to get rid of traditional understanding" — golden signals remain "essential"; the gen AI metrics are an **additional layer** (cost, safety, quality) on top. - She frames the shift as an expanded engineering responsibility: engineers must now "care about additional layer of complexity" beyond just shipping and running applications. - **Commercial context (stated in talk):** She promotes Datadog's "agent observability" product, mentions a booth, free trial QR codes, and invites attendees to contact her. --- ## Source Notes - All claims above are attributed solely to the spoken transcript; no curated speaker metadata beyond the title was supplied beyond the transcript's self-identification. - Some model/number names in the transcript appear garbled or uncertain ("Hi Cool 4.5," "Opus 4.8," context window "blow to 132"); these are recorded as expressed, with likely intended readings noted only as such. - Unresolved tension preserved: she calls quality "arguably the most crucial dimension" while also framing safety as "zero tolerance" — the talk does not rank the three layers against each other definitively. # Speaker Roles & Provenance Record ## Video Identification - **Title:** *Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog* - **Video ID:** `01a15ca7-1e61-77bb-8a0d-e8ebf65c6c1d` ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **Marina Petzel** | Presenter; self-identified **"GenAI advocate at Datadog"** | Sole speaker throughout; introduces herself twice ("my name is Marina Petzel"; closing repeats name, rendered "Marina Betzel" in transcript — likely transcription variance) | No guests, co-presenters, or interviewers appear. The format is a conference-style talk (audience greeting, closing contact invitation). ## Speaker Affiliation & Promotion ⚠️ - The speaker represents **Datadog**, and the closing segment contains a **promotional block** for Datadog's **"agent observability" product**: - Mentions a **conference booth**, **QR codes for free trials**, and "really good offers" to try the product. - Speaker invites attendees to contact her afterwards (personal contact QR code). - All product claims (e.g., product "allows you to quickly and easy understand if your AI application is healthy") originate from a vendor employee and should be treated as **promotional/vendor-sourced**, not independent evaluation. ## Referenced Sources & Third-Party Claims - **Internal research claim:** "what we will see from um our uh own research that 70% of the spend sometimes are redundant" — attributed to Datadog's own research; methodology not given. - **Named third-party products/providers (as examples only):** "Hi Cool 4.5" → "Opus 4.8" (apparent garbled references to Claude model tiers); "GPT-5.5"; providers "OpenAI or Tropic" (likely "Anthropic" — transcription variance). - **General claims without cited sources:** 8x cost increase from context window growth; Opus ~15x higher cost per request. No external citations, papers, or links are provided in the transcript. - **Techniques referenced (no specific sources):** BERT / embedding similarity, regex / named-entity recognition, LLM-as-a-judge, top-K accuracy, NDCG, NPS. ## Comment Availability - No information in the transcript about comments, Q&A availability, or audience interaction recording. **Unknown.** ## Provenance Notes / Uncertainties - Transcript contains multiple apparent transcription errors ("Hi Cool 4.5", "Tropic", "let scores"/"lets metrics" for "LLM", name spelled both "Petzel" and "Betzel"). Flagged as variance; title metadata ("Marina Petzel") treated as authoritative for the speaker's name. - No curated metadata supplied beyond the title; speaker identity rests on self-introduction in the transcript. - Model names and cost figures are **unverified claims from the speaker**, not established facts. \>\> Hello everyone. Thank you for joining me today. It's time to begin. And for years, many of us monitored our applications with the golden signals, which is was latency, error, traffic, and saturation. And shortly we call them lads. And they were and still they are and will be remain sufficient and critical for determine the health of our applications. However, today when we enter the new era era of generative AI, it's becoming not enough. And today we going to talk about what are the new metrics we need to look if we want to build the healthy gen AI application. Very shortly, my name is Marina Petzel. I am gen AI advocate here in DataDog. And today I will walk you through the metrics that we need to look for. So, when we talk about traditional applications, the main specific of them is that the same input always yield the same output, right? It's predictable. But with generative AI applications, we all know that the same prompt can generate different responses every time we run it. And which means we cannot rely on the standard regression testing alone. And you must constantly evaluate quality in the live environment. The second difference between traditional applications and gen AI applications, it's variable cost structure. Traditional compute cost are predictable, right? We pay for machine that is running our code. But for gen AI, this this is dynamic variable here. They might They vary from the number of tokens, the model that we are using, or the size of the context window that you have. And it's become essential to do the real-time granular course cost to prevent the spending out of control. The third problem is that we face the new attack vectors. We used to worry about SQL injection or cross-site scripting before, but today we must defend against the prompt injection, jailbreaking, or PII outage in our output. And that's becoming a big problem, and this are security issue, not a simple 500 error code can will never catch, and demanded entirely new forms of security monitoring. And finally, quality is very subjective for GenAI applications. The traditional systems either work or do not work, right? But generative AI output is judged on the spectrum. Is it relevant? Is it accurate? Is it complete or not? And the problem is that 200 okay response from your server doesn't necessarily mean that the answer was helpful or correct for your final user. And we must now integrate the continuous quality evaluation directly in your monitoring stack. And at the beginning we all were thinking that these golden signals will be enough for our systems, but unfortunately they are not. Traditional metrics will tell you if your system is up and running, but it won't tell you how well it's running. And established metrics lets will be still essential, but we going to talk about what else on addition to that we need to implement into our systems? And first I want to start with the cost monitoring. How we can track the spending before it uh spirals out of control? And I think we all know the financial implications of GenAI, and a lot of companies are facing the huge outage of the uh cost per their monthly usage. And while the same is true as for traditional applications, there were more linear scale between the throughput and the uh the cost. If the user loads um load goes up, our resources uh usage goes up, and sequentially the cost goes up. But the cost structure for LLM is highly dynamic and unpredictable, which means we can we don't have this visibility, and we need to really pay attention how we track that. And what are the problems you might encounter with the cost? And here we talk about the three major way for cost tracking, uh and so you can track them um to prevent the cost outage. So, the first is a token creep. Uh this happens when engineers or product managers trying to improve the quality quietly increasing the context window. And let's say the context window was 4,000 tokens, and then it's like blow to 132 without any financial review. And what it can cost like the the cost is depends very uh significantly to the context count, and that might result up to eight x cost increased. And and a month it might cost you thousands of dollars. Second is the model drift. Uh that is also the pretty typical problem when the team switches from cheaper and faster model, like let's say Hi Cool 4.5, to the more powerful and more expensive model, like Opus 4.8. And which promise you obviously the better quality. But at the same time, Opus, for example, uh can be higher 15 times higher uh per the cost request. And this change in model, even with the same usage volume, might blow your cost significantly within a month, too. And finally, it's on hash code. In generative AI, the same query can hit the API repeatedly if you haven't implemented the effective caching layer. And what you will see from um our uh own research that 70% of the spend sometimes are redundant because the system is paying for exactly the same expensive model completion uh over and over again. And with this granular visibility into these three layers, you would be able to escalate your cost rapidly. And now, let's talk about how you can prevent that and get more granular data. And when we um talk about that, we uh try to think about the implementation of cost attribution strategy, which is you must tag absolutely everything. For example, uh tagging uh can help you to uh Why it's important, right? You receive a lot of uh interactions with your agents, and but all of these uh interactions are undifferentiated. You don't know what's blow your actual cost. That is why first layer what you can implement is the feature level level. By tagging the intention of the feature, like it was a chat or summarization, you can attribute the cost for the specific product areas. And this data is crucial not only for engineering your cost management, but also for the product road map decisions, like knowing which features are actually the budget hawks. The second level is the use user level. Using the tags like the user ID or your org name is essential for chargebacks, accurate billing, but also quickly identify by the patterns of abusive usage before they drain your budget. Third is the model level. Also tag your models like GPT-5.5 or the provider, OpenAI or Tropic. It will give you visibility into the high-level areas of cost optimization. And you can benchmark the true expenses of using one model versus another. And the finally, endpoint level tagging. Attributing cost to regions or to your production or like to your environment like production or staging helps your infrastructure planning. And you can see how regional distribution or staging versus production environment can impact your total spend. And by making this tagging mandatory practice, you gain the essential visibility to understand exactly where your money is going and how important to control it. Well, but now let's talk if like let's let's imagine that we handle all the cost issues, right? We took that under control. And your finance team is obviously get very happy, but the next one problem is the safety. And your AI can be cheap, but still leak some of the sensitive or personal information, which is not great. And even though like your let scores can be perfect, it cannot tell you if there are some metrics went wrong. So, what we can track here in the safety metrics to First is the prompt injection rate, which has become the very crucial problem. This metrics detects the manipulation attempts into your user prompts. There are It's infrastructure If the infrastructure design doesn't have any guardrails. And this can be um prevent by using some advanced techniques like a pattern matching or specialized classifier models. And for this, we usually need to target for very aggressive thresholds. The second is PII detection rate. Um PII So, which is a crucial and zero tolerance issue here. We can scan all the model outputs for sensitive data like social security, like credit cards, or personal information. And you can use some um methods like regex or named entity recognition here. And only acceptable target should be 0% of your production outputs. The third is the content moderation score. How you can determine if your content is toxic, harmful, or biased uh towards some of the topics. And by running all outputs through toxicity classifier can give give you a scores uh how what is the risk of your output for being exposed to this uh toxic uh output. And this allows you to set some clear thresholds, and we always target to uh minimize that as much as we can. And then finally, the jailbreak attempts. And there are persistent attempts by users to bypass the safety guardrails and often um through the elaborate scenarios. And tracking tracking patterns to indicate the system prompt overrides, you should design and become so it become the absolute blocker for you. So, ideally, we should block all the 100% of the attempts of that. Now, let's imagine that cost is checked, security is checked, but now you arrive to arguably the most crucial dimension and the quality monitor that at least the most crucial for our final users. And even though your let's metrics showing that your service is up and running and running, your cost and safety tells you that your service is budget compliant and also secure, but ultimately for any application, but more for generative application, it's whenever output is good. But we all know that 200 okay response from your server sometimes can be a lie if the answer is hallucination. And I'm sure many of you got some response from modern AI applications, right? But like we know that it's can confidently by like but lie to you. And now let's understand what type of metrics you can apply implement to your application right now to prevent it from being unsafe for your final user. We recommend to focus on these five metrics. One of them is hallucination rate. Um, this measures the percentage of respondent contain claims unsupported by the grounded data. Um, we need to mix that with the manual review as well as with automated fact checks to achieve the aggressive targets here, too. The final The next one is the relevance scores, which helps you to understand if the response actually address the user question. Um, that techniques like that you can use here is similarity using bird or embedding similarity that can score you um, how relevant is the final answer was to the user. The next one is user satisfaction, which directly falls into your end user. Uh, you must implement You can implement some mechanisms like thumbs up, thumbs down, or some of the stars ranking, and also NPS scores um, to capture the subjective data, but also you should target that you get more than 80 85% of the positive feedback, at least. Next one is the answer completeness, and this metric is often evaluated by LLM as a judge, answering whenever the response was fully or holistically addressed um, the user intent. Um, and final one, the fifth one, um, if you are using rag, retrieval augmented generation system, you should always track the quality of your rag. What's if all the documents that were fetched were relevant to the user query? And here you have a lot of metrics and mechanisms how you can calculate that. For example, top K accuracy or normalized discounted gains, and that is uh, essential to track because whenever like you contacts is explode is getting harder and harder to pick the most relevant one. And by implementing this metrics, you can gain some understanding of like more dimensional view of the quality that necessary you need to optimize to improve your journey application. Well, and now let's summarize um what we need to implement to make sure that our journey application is healthy, right? We are not saying that we need to get rid of traditional understanding. We always need to look into our metrics, lay let's latency error traffic and saturation, but also on top of that, we need to put another layer. Layer of which is more relevant to generative applications. And in this case, we talked about the cost, safety, and also quality. And I know that for us as an engineer is becoming more and more critical right now to track it all not be don't need just build application and make sure that it's running in production, but now we also need to care about additional layer of complexity. And thankfully, Datadog has your back here since we are working in observability and monitoring space for for really long time. So, we do have our agent observability uh product which allows you to quickly and easy understand if your AI application is healthy, if it's running, or there is any problem that you need to fix. And we do have a lot of different way things that you can track inside of that. And um if you want to hear more about that, please feel free to see me afterwards. We also do have the booth here uh which you can check it out. We do have a really good um offers for you if you want to just try it out and see how agent observability can help your application. We have some QR codes for the free trials. And with that, I want to thank everyone. Again, my name is Marina Betzel. If you want to contact me, please feel free to use this QR code. And yeah, thank you for being here today. \>\> \[music\]