--- schema: 1 type: video title: >- Production Evals For Agentic AI Systems - Nishant Gupta, Meta Superintelligence Labs description: >- As AI systems evolve from chat interfaces into autonomous agents capable of reasoning, planning, and tool usage, traditional evaluation approaches are breaking down. Offline benchmarks and static datasets fail to capture the complexity, non-determinism, and operational risks of real-world AI systems operating in production environments. In this talk, I’ll share practical lessons and architectural patterns for building evaluation systems for agentic AI workflows at scale. We’ll explore how modern AI platforms are shifting from one-time benchmark testing toward continuous evaluation pipelines integrated directly into production infrastructure. Topics include: - Why offline evals fail for autonomous AI systems - Evaluating tool use, planning, reasoning, and multi-step workflows - Online vs offline eval architectures - Human-in-the-loop evaluation systems - Detecting drift, hallucinations, and unsafe behaviors - Building feedback loops for continuous improvement - Observability and telemetry for agentic workflows - Reliability metrics beyond model accuracy Attendees will leave with practical frameworks for designing scalable evaluation systems capable of measuring real-world AI behavior, reliability, and operational impact. Speakers: - Nishant Gupta (Meta Superintelligence Labs): Nishant Gupta is a Software Engineering Tech Lead at Meta Superintelligence Labs focused on building the training and inference AI Infrastructure. LinkedIn: https://www.linkedin.com/in/nishantgupta-ai/ GitHub: https://github.com/nishantgpt-lab canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/production-evals-for-agentic-ai-systems-nishant-gup-f4556ad9252b markdown: >- https://liquidgold.cc/channel/aidotengineer/video/production-evals-for-agentic-ai-systems-nishant-gup-f4556ad9252b.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T09:19:43.251Z' language: null children: [] videoId: vljxQZfJ9wY durationSeconds: 492 publishedAt: '2026-06-25T18:15:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 15 endSeconds: 124 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=15' heldReason: rights - ordinal: 1 text: null startSeconds: 15 endSeconds: 124 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=15' heldReason: rights - ordinal: 2 text: null startSeconds: 135 endSeconds: 228 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=135' heldReason: rights - ordinal: 3 text: null startSeconds: 135 endSeconds: 228 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=135' heldReason: rights - ordinal: 4 text: null startSeconds: 135 endSeconds: 228 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=135' heldReason: rights - ordinal: 5 text: null startSeconds: 228 endSeconds: 238 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=228' heldReason: rights - ordinal: 6 text: null startSeconds: 238 endSeconds: 327 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=238' heldReason: rights - ordinal: 7 text: null startSeconds: 238 endSeconds: 327 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=238' heldReason: rights - ordinal: 8 text: null startSeconds: 238 endSeconds: 327 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=238' heldReason: rights - ordinal: 9 text: null startSeconds: 338 endSeconds: 448 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=338' heldReason: rights - ordinal: 10 text: null startSeconds: 338 endSeconds: 448 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=338' heldReason: rights - ordinal: 11 text: null startSeconds: 338 endSeconds: 448 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=338' heldReason: rights - ordinal: 12 text: null startSeconds: 448 endSeconds: 459 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=448' heldReason: rights - ordinal: 13 text: null startSeconds: 459 endSeconds: 492 watch: 'https://liquidgold.cc/watch/vljxQZfJ9wY?t=459' heldReason: rights --- # Production Evals For Agentic AI Systems - Nishant Gupta, Meta Superintelligence Labs As AI systems evolve from chat interfaces into autonomous agents capable of reasoning, planning, and tool usage, traditional evaluation approaches are breaking down. Offline benchmarks and static datasets fail to capture the complexity, non-determinism, and operational risks of real-world AI systems operating in production environments. In this talk, I’ll share practical lessons and architectural patterns for building evaluation systems for agentic AI workflows at scale. We’ll explore how modern AI platforms are shifting from one-time benchmark testing toward continuous evaluation pipelines integrated directly into production infrastructure. Topics include: - Why offline evals fail for autonomous AI systems - Evaluating tool use, planning, reasoning, and multi-step workflows - Online vs offline eval architectures - Human-in-the-loop evaluation systems - Detecting drift, hallucinations, and unsafe behaviors - Building feedback loops for continuous improvement - Observability and telemetry for agentic workflows - Reliability metrics beyond model accuracy Attendees will leave with practical frameworks for designing scalable evaluation systems capable of measuring real-world AI behavior, reliability, and operational impact. Speakers: - Nishant Gupta (Meta Superintelligence Labs): Nishant Gupta is a Software Engineering Tech Lead at Meta Superintelligence Labs focused on building the training and inference AI Infrastructure. LinkedIn: https://www.linkedin.com/in/nishantgupta-ai/ GitHub: https://github.com/nishantgpt-lab Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/production-evals-for-agentic-ai-systems-nishant-gup-f4556ad9252b · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:03 to 2:04) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (2:04 to 3:48) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (3:48 to 5:27) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (5:27 to 7:28) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (7:28 to 8:12) > Held: transcript rights do not permit this chunk's text.