--- schema: 1 type: video title: 'Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog' description: >- Latency, errors, traffic and saturation tell you your app is up. They don't tell you if it's right. Marina Petzel, GenAI advocate at Datadog, explains why the classic golden signals are necessary but no longer enough for generative AI applications, and which metrics to add on top. She covers cost monitoring (token creep, model drift, uncached calls and tagging for cost attribution), safety monitoring (prompt injection, PII leakage, toxicity and jailbreaks) and quality monitoring (hallucination rate, relevance, user satisfaction, completeness and RAG retrieval quality). In this talk: • Four ways GenAI apps break traditional monitoring assumptions • Three hidden cost drivers and a four-level tagging strategy • Safety metrics and the thresholds to aim for • Five quality metrics, from hallucination rate to RAG retrieval quality SPEAKER Marina Petzel, GenAI Advocate, Datadog LinkedIn: https://www.linkedin.com/in/marina-petzel/ LINKS Datadog Agent Observability: https://www.datadoghq.com/products/ai/agent-observability/ LLM Observability docs: https://docs.datadoghq.com/llm_observability/ Datadog: https://www.datadoghq.com CHAPTERS 0:00 Intro 0:12 Golden signals aren't enough anymore 1:12 How GenAI apps are different 2:22 New attack vectors 2:47 Quality is subjective 3:51 Cost monitoring 4:56 Token creep, model drift, uncached calls 6:55 Cost attribution: tag everything 9:15 Safety metrics 9:55 Prompt injection, PII, toxicity, jailbreaks 12:09 Quality monitoring 13:19 Five quality metrics 15:43 Putting it all together 16:38 Datadog Agent Observability Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMObservability #GenAI #AIEngineer canonical: 'https://liquidgold.cc/watch/rTojoVotlD8' markdown: 'https://liquidgold.cc/watch/rTojoVotlD8.md' llms: 'https://liquidgold.cc/watch/rTojoVotlD8/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:20:08.814Z' language: null children: [] videoId: rTojoVotlD8 durationSeconds: 1081 publishedAt: '2026-10-03T01:30:33.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 23 endSeconds: 80 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=23' heldReason: rights - ordinal: 1 text: null startSeconds: 80 endSeconds: 90 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=80' heldReason: rights - ordinal: 2 text: null startSeconds: 90 endSeconds: 144 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=90' heldReason: rights - ordinal: 3 text: null startSeconds: 144 endSeconds: 155 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=144' heldReason: rights - ordinal: 4 text: null startSeconds: 155 endSeconds: 168 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=155' heldReason: rights - ordinal: 5 text: null startSeconds: 168 endSeconds: 178 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=168' heldReason: rights - ordinal: 6 text: null startSeconds: 178 endSeconds: 235 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=178' heldReason: rights - ordinal: 7 text: null startSeconds: 235 endSeconds: 247 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=235' heldReason: rights - ordinal: 8 text: null startSeconds: 247 endSeconds: 306 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=247' heldReason: rights - ordinal: 9 text: null startSeconds: 306 endSeconds: 318 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=306' heldReason: rights - ordinal: 10 text: null startSeconds: 318 endSeconds: 421 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=318' heldReason: rights - ordinal: 11 text: null startSeconds: 421 endSeconds: 431 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=421' heldReason: rights - ordinal: 12 text: null startSeconds: 431 endSeconds: 561 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=431' heldReason: rights - ordinal: 13 text: null startSeconds: 561 endSeconds: 571 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=561' heldReason: rights - ordinal: 14 text: null startSeconds: 571 endSeconds: 604 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=571' heldReason: rights - ordinal: 15 text: null startSeconds: 604 endSeconds: 615 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=604' heldReason: rights - ordinal: 16 text: null startSeconds: 615 endSeconds: 732 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=615' heldReason: rights - ordinal: 17 text: null startSeconds: 732 endSeconds: 744 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=732' heldReason: rights - ordinal: 18 text: null startSeconds: 744 endSeconds: 810 watch: 'https://liquidgold.cc/watch/rTojoVotlD8?t=744' heldReason: rights --- # Your LLM App Returned 200 OK. It Was Still Wrong. — Marina Petzel, Datadog Latency, errors, traffic and saturation tell you your app is up. They don't tell you if it's right. Marina Petzel, GenAI advocate at Datadog, explains why the classic golden signals are necessary but no longer enough for generative AI applications, and which metrics to add on top. She covers cost monitoring (token creep, model drift, uncached calls and tagging for cost attribution), safety monitoring (prompt injection, PII leakage, toxicity and jailbreaks) and quality monitoring (hallucination rate, relevance, user satisfaction, completeness and RAG retrieval quality). In this talk: • Four ways GenAI apps break traditional monitoring assumptions • Three hidden cost drivers and a four-level tagging strategy • Safety metrics and the thresholds to aim for • Five quality metrics, from hallucination rate to RAG retrieval quality SPEAKER Marina Petzel, GenAI Advocate, Datadog LinkedIn: https://www.linkedin.com/in/marina-petzel/ LINKS Datadog Agent Observability: https://www.datadoghq.com/products/ai/agent-observability/ LLM Observability docs: https://docs.datadoghq.com/llm_observability/ Datadog: https://www.datadoghq.com CHAPTERS 0:00 Intro 0:12 Golden signals aren't enough anymore 1:12 How GenAI apps are different 2:22 New attack vectors 2:47 Quality is subjective 3:51 Cost monitoring 4:56 Token creep, model drift, uncached calls 6:55 Cost attribution: tag everything 9:15 Safety metrics 9:55 Prompt injection, PII, toxicity, jailbreaks 12:09 Quality monitoring 13:19 Five quality metrics 15:43 Putting it all together 16:38 Datadog Agent Observability Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMObservability #GenAI #AIEngineer Canonical: https://liquidgold.cc/watch/rTojoVotlD8 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:12 to 2:24) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (2:24 to 3:55) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (3:55 to 5:06) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (5:06 to 7:01) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (7:01 to 9:21) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (9:21 to 12:12) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (12:12 to 13:30) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (13:30 to 15:47) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 8: Document excerpt (15:47 to 18:01) > Held: transcript rights do not permit this chunk's text.