--- schema: 1 type: video title: 'Build Evals That Actually Matter - Nick Ung & Akshay Sharma, Lyft' description: >- Your agent passes offline evals at 90%. You ship. Production immediately finds failure modes your eval never saw. Sound familiar? The culprit is almost always the same: the "customer" in your offline eval is an off-the-shelf LLM that sounds nothing like your real users, and your synthetic test set doesn't capture how messy, angry, or off-topic real conversations get. Your eval was too easy. At Lyft, our customer-care agents resolve roughly a third of all customer issues — millions of conversations a month. To trust them at that scale, we built an adversarial user simulator: a fine-tuned LLM trained on real Lyft rider and driver transcripts that can role-play frustrated, confused, and adversarial users with the same distribution as production. It found regressions our synthetic dataset missed for months. This talk walks through the full eval lifecycle that surrounds it: the harness primitives that let any engineer write a benchmark in 20 lines, how we calibrate LLM-judge rubrics against human labels until they match inter-rater agreement, how we route failed production traces back into the offline test set, and the continual-learning loop that feeds improvements into prompts, harness, and the model. Slides: - https://docs.google.com/presentation/d/14WuuV6PqUXgXLBVWuZmwsYhmI-lauoNuYyFGvlzHQc4/edit?usp=sharing Speakers: - Nick Ung (Lyft): Nick Ung leads Data Science for Safety & Customer Care at Lyft, where his team built and operates the multi-agent platform that powers AI agents resolving roughly a third of all Lyft customer issues. - LinkedIn: https://www.linkedin.com/in/unglikteng - Akshay Sharma (Lyft): Akshay Sharma is a Senior ML Engineer at Lyft and tech lead of the Lyft’s Customer Care Agent Architecture and the Builder Platform, which he co-founded with Nick. - LinkedIn: https://www.linkedin.com/in/akshay-sharma-1995/ --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/3z2uT5aDx_Y-build-evals-that-actually-matter-nick-ung-akshay-sharma-lyft Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/build-evals-that-actually-matter-nick-ung-akshay-sh-b9a937c1f87a markdown: >- https://liquidgold.cc/channel/aidotengineer/video/build-evals-that-actually-matter-nick-ung-akshay-sh-b9a937c1f87a.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T07:18:57.566Z' language: null children: [] videoId: 3z2uT5aDx_Y durationSeconds: 2265 publishedAt: '2026-07-19T12:15:06.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 140 endSeconds: 242 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=140' heldReason: rights - ordinal: 1 text: null startSeconds: 242 endSeconds: 253 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=242' heldReason: rights - ordinal: 2 text: null startSeconds: 253 endSeconds: 375 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=253' heldReason: rights - ordinal: 3 text: null startSeconds: 375 endSeconds: 386 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=375' heldReason: rights - ordinal: 4 text: null startSeconds: 386 endSeconds: 467 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=386' heldReason: rights - ordinal: 5 text: null startSeconds: 479 endSeconds: 604 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=479' heldReason: rights - ordinal: 6 text: null startSeconds: 616 endSeconds: 699 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=616' heldReason: rights - ordinal: 7 text: null startSeconds: 711 endSeconds: 791 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=711' heldReason: rights - ordinal: 8 text: null startSeconds: 803 endSeconds: 915 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=803' heldReason: rights - ordinal: 9 text: null startSeconds: 926 endSeconds: 1015 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=926' heldReason: rights - ordinal: 10 text: null startSeconds: 1026 endSeconds: 1116 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=1026' heldReason: rights - ordinal: 11 text: null startSeconds: 1206 endSeconds: 1216 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=1206' heldReason: rights - ordinal: 12 text: null startSeconds: 1305 endSeconds: 1316 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=1305' heldReason: rights - ordinal: 13 text: null startSeconds: 1419 endSeconds: 1508 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=1419' heldReason: rights - ordinal: 14 text: null startSeconds: 1508 endSeconds: 1519 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=1508' heldReason: rights - ordinal: 15 text: null startSeconds: 1651 endSeconds: 1754 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=1651' heldReason: rights - ordinal: 16 text: null startSeconds: 1754 endSeconds: 1765 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=1754' heldReason: rights - ordinal: 17 text: null startSeconds: 1867 endSeconds: 1877 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=1867' heldReason: rights - ordinal: 18 text: null startSeconds: 2005 endSeconds: 2142 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=2005' heldReason: rights - ordinal: 19 text: null startSeconds: 2142 endSeconds: 2154 watch: 'https://liquidgold.cc/watch/3z2uT5aDx_Y?t=2142' heldReason: rights --- # Build Evals That Actually Matter - Nick Ung & Akshay Sharma, Lyft Your agent passes offline evals at 90%. You ship. Production immediately finds failure modes your eval never saw. Sound familiar? The culprit is almost always the same: the "customer" in your offline eval is an off-the-shelf LLM that sounds nothing like your real users, and your synthetic test set doesn't capture how messy, angry, or off-topic real conversations get. Your eval was too easy. At Lyft, our customer-care agents resolve roughly a third of all customer issues — millions of conversations a month. To trust them at that scale, we built an adversarial user simulator: a fine-tuned LLM trained on real Lyft rider and driver transcripts that can role-play frustrated, confused, and adversarial users with the same distribution as production. It found regressions our synthetic dataset missed for months. This talk walks through the full eval lifecycle that surrounds it: the harness primitives that let any engineer write a benchmark in 20 lines, how we calibrate LLM-judge rubrics against human labels until they match inter-rater agreement, how we route failed production traces back into the offline test set, and the continual-learning loop that feeds improvements into prompts, harness, and the model. Slides: - https://docs.google.com/presentation/d/14WuuV6PqUXgXLBVWuZmwsYhmI-lauoNuYyFGvlzHQc4/edit?usp=sharing Speakers: - Nick Ung (Lyft): Nick Ung leads Data Science for Safety & Customer Care at Lyft, where his team built and operates the multi-agent platform that powers AI agents resolving roughly a third of all Lyft customer issues. - LinkedIn: https://www.linkedin.com/in/unglikteng - Akshay Sharma (Lyft): Akshay Sharma is a Senior ML Engineer at Lyft and tech lead of the Lyft’s Customer Care Agent Architecture and the Builder Platform, which he co-founded with Nick. - LinkedIn: https://www.linkedin.com/in/akshay-sharma-1995/ --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/3z2uT5aDx_Y-build-evals-that-actually-matter-nick-ung-akshay-sharma-lyft Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/build-evals-that-actually-matter-nick-ung-akshay-sh-b9a937c1f87a · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:05 to 2:09) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (2:09 to 4:02) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (4:02 to 6:15) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (6:15 to 7:47) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (7:47 to 10:04) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (10:04 to 11:39) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (11:39 to 13:11) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (13:11 to 15:15) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 8: Document excerpt (15:15 to 16:55) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 9: Document excerpt (16:55 to 18:36) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 10: Document excerpt (18:36 to 20:06) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 11: Document excerpt (20:06 to 21:45) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 12: Document excerpt (21:45 to 23:28) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 13: Document excerpt (23:28 to 25:08) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 14: Document excerpt (25:08 to 27:20) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 15: Document excerpt (27:20 to 29:14) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 16: Document excerpt (29:14 to 31:07) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 17: Document excerpt (31:07 to 33:13) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 18: Document excerpt (33:13 to 35:42) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 19: Document excerpt (35:42 to 37:45) > Held: transcript rights do not permit this chunk's text.