--- schema: 1 type: video title: 'LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize' description: >- Your agent called tool B before tool A, and B has a dependency on A. You did not catch it because nothing in your code audits agents. The telemetry does. Dat from Arize AI walks through what observability actually means when the system you are debugging is nondeterministic and the execution path changes with every run. The talk covers the five flavors of eval signal (LLM as judge, human feedback, golden datasets, deterministic checks, business metrics), what scope to run them at (single span, multispan, trajectory, session), and where this is heading. Arize Phoenix is open source, runs as a single container, no Kubernetes required. The enterprise product adds an AI layer called Alex that scans traces, surfaces high latency and errors, and creates evals automatically. The stated goal: automate you out of the observability loop entirely. Speaker info: - https://www.linkedin.com/in/datdarylngo/ - https://x.com/dat_attacked --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/JsCCrBF7F1g-llm-observability-evaluation-experimentation-platform-dat-ngo-arize-ai Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/JsCCrBF7F1g' markdown: 'https://liquidgold.cc/watch/JsCCrBF7F1g.md' llms: 'https://liquidgold.cc/watch/JsCCrBF7F1g/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T09:39:52.006Z' language: null children: [] videoId: JsCCrBF7F1g durationSeconds: 992 publishedAt: '2026-06-07T18:00:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 115 endSeconds: 247 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=115' heldReason: rights - ordinal: 1 text: null startSeconds: 115 endSeconds: 247 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=115' heldReason: rights - ordinal: 2 text: null startSeconds: 258 endSeconds: 337 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=258' heldReason: rights - ordinal: 3 text: null startSeconds: 258 endSeconds: 337 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=258' heldReason: rights - ordinal: 4 text: null startSeconds: 337 endSeconds: 349 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=337' heldReason: rights - ordinal: 5 text: null startSeconds: 349 endSeconds: 485 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=349' heldReason: rights - ordinal: 6 text: null startSeconds: 349 endSeconds: 485 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=349' heldReason: rights - ordinal: 7 text: null startSeconds: 349 endSeconds: 485 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=349' heldReason: rights - ordinal: 8 text: null startSeconds: 349 endSeconds: 485 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=349' heldReason: rights - ordinal: 9 text: null startSeconds: 495 endSeconds: 593 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=495' heldReason: rights - ordinal: 10 text: null startSeconds: 495 endSeconds: 593 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=495' heldReason: rights - ordinal: 11 text: null startSeconds: 604 endSeconds: 693 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=604' heldReason: rights - ordinal: 12 text: null startSeconds: 604 endSeconds: 693 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=604' heldReason: rights - ordinal: 13 text: null startSeconds: 705 endSeconds: 793 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=705' heldReason: rights - ordinal: 14 text: null startSeconds: 705 endSeconds: 793 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=705' heldReason: rights - ordinal: 15 text: null startSeconds: 705 endSeconds: 793 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=705' heldReason: rights - ordinal: 16 text: null startSeconds: 804 endSeconds: 893 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=804' heldReason: rights - ordinal: 17 text: null startSeconds: 904 endSeconds: 992 watch: 'https://liquidgold.cc/watch/JsCCrBF7F1g?t=904' heldReason: rights --- # LLM Observability, Evaluation, Experimentation Platform — Dat Ngo, Arize Your agent called tool B before tool A, and B has a dependency on A. You did not catch it because nothing in your code audits agents. The telemetry does. Dat from Arize AI walks through what observability actually means when the system you are debugging is nondeterministic and the execution path changes with every run. The talk covers the five flavors of eval signal (LLM as judge, human feedback, golden datasets, deterministic checks, business metrics), what scope to run them at (single span, multispan, trajectory, session), and where this is heading. Arize Phoenix is open source, runs as a single container, no Kubernetes required. The enterprise product adds an AI layer called Alex that scans traces, surfaces high latency and errors, and creates evals automatically. The stated goal: automate you out of the observability loop entirely. Speaker info: - https://www.linkedin.com/in/datdarylngo/ - https://x.com/dat_attacked --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/JsCCrBF7F1g-llm-observability-evaluation-experimentation-platform-dat-ngo-arize-ai Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/JsCCrBF7F1g · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.