--- schema: 1 type: video title: >- AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash description: >- The people annotating DoorDash's eval data are not engineers, and they build their own annotation tools. Because the GenAI platform team went API first, strategy and operations staff can point a coding agent at those endpoints and vibe code whatever interface their use case needs, whether that is grading restaurant menus or reviewing images. The platform team stopped trying to anticipate every UI, and shipped stable APIs instead. Nachiket Paranjape and Swaroop Chitlur Haridas make the broader case that evals stopped being an engineering harness for them and became a cross functional job. That reframing has an org chart attached. Strategy and operations set the quality bar, product managers turn it into rubrics, operations run the annotations, and engineering supplies telemetry, datasets and judges. Which group actually owns a judge prompt varies by team, and they treat that variation as a sign the org is still learning rather than a problem to standardize away. The loop underneath is deliberately plain: trace, sample down to something a human will really look at, annotate, promote a golden set, calibrate the judge against it, then monitor and go again. Judge calibration runs self serve through a UI, showing the original and optimized prompts side by side so a product manager can see what changed and decide whether to trust it. Per annotation cost fell sharply. Speaker info: Nachiket Paranjape: - https://x.com/nmparanjape - https://www.linkedin.com/in/nachiketparanjape/ Timestamps: 0:00 - The GenAI platform team, and its three forces 2:05 - Why eval became the fourth pillar 3:05 - UI first, then API first, then workflow first 4:01 - Evals as a team sport, not an engineering harness 4:57 - Who owns which part of quality 5:53 - The continuous loop: trace, sample, annotate, calibrate 7:42 - Telemetry and workflow as two surfaces 9:32 - Operators vibe coding their own annotation UIs 11:21 - Calibrating judge prompts, self serve 13:10 - Different teams, different prompt owners 14:04 - What it did to annotation cost --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/bMjlRrWjdT0-youtube-de0e357f07f3800aaed9 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: 'https://liquidgold.cc/watch/bMjlRrWjdT0' markdown: 'https://liquidgold.cc/watch/bMjlRrWjdT0.md' llms: 'https://liquidgold.cc/watch/bMjlRrWjdT0/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-24T04:28:52.538Z' language: null children: [] videoId: bMjlRrWjdT0 durationSeconds: 971 publishedAt: '2026-08-28T15:00:03.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 25 endSeconds: 125 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=25' heldReason: rights - ordinal: 1 text: null startSeconds: 25 endSeconds: 125 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=25' heldReason: rights - ordinal: 2 text: null startSeconds: 25 endSeconds: 125 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=25' heldReason: rights - ordinal: 3 text: null startSeconds: 25 endSeconds: 125 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=25' heldReason: rights - ordinal: 4 text: null startSeconds: 136 endSeconds: 194 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=136' heldReason: rights - ordinal: 5 text: null startSeconds: 136 endSeconds: 194 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=136' heldReason: rights - ordinal: 6 text: null startSeconds: 136 endSeconds: 194 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=136' heldReason: rights - ordinal: 7 text: null startSeconds: 194 endSeconds: 204 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=194' heldReason: rights - ordinal: 8 text: null startSeconds: 204 endSeconds: 247 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=204' heldReason: rights - ordinal: 9 text: null startSeconds: 258 endSeconds: 304 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=258' heldReason: rights - ordinal: 10 text: null startSeconds: 316 endSeconds: 353 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=316' heldReason: rights - ordinal: 11 text: null startSeconds: 365 endSeconds: 469 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=365' heldReason: rights - ordinal: 12 text: null startSeconds: 479 endSeconds: 572 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=479' heldReason: rights - ordinal: 13 text: null startSeconds: 572 endSeconds: 584 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=572' heldReason: rights - ordinal: 14 text: null startSeconds: 584 endSeconds: 684 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=584' heldReason: rights - ordinal: 15 text: null startSeconds: 584 endSeconds: 684 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=584' heldReason: rights - ordinal: 16 text: null startSeconds: 584 endSeconds: 684 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=584' heldReason: rights - ordinal: 17 text: null startSeconds: 695 endSeconds: 793 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=695' heldReason: rights - ordinal: 18 text: null startSeconds: 695 endSeconds: 793 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=695' heldReason: rights - ordinal: 19 text: null startSeconds: 793 endSeconds: 803 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=793' heldReason: rights - ordinal: 20 text: null startSeconds: 860 endSeconds: 971 watch: 'https://liquidgold.cc/watch/bMjlRrWjdT0?t=860' heldReason: rights --- # AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash The people annotating DoorDash's eval data are not engineers, and they build their own annotation tools. Because the GenAI platform team went API first, strategy and operations staff can point a coding agent at those endpoints and vibe code whatever interface their use case needs, whether that is grading restaurant menus or reviewing images. The platform team stopped trying to anticipate every UI, and shipped stable APIs instead. Nachiket Paranjape and Swaroop Chitlur Haridas make the broader case that evals stopped being an engineering harness for them and became a cross functional job. That reframing has an org chart attached. Strategy and operations set the quality bar, product managers turn it into rubrics, operations run the annotations, and engineering supplies telemetry, datasets and judges. Which group actually owns a judge prompt varies by team, and they treat that variation as a sign the org is still learning rather than a problem to standardize away. The loop underneath is deliberately plain: trace, sample down to something a human will really look at, annotate, promote a golden set, calibrate the judge against it, then monitor and go again. Judge calibration runs self serve through a UI, showing the original and optimized prompts side by side so a product manager can see what changed and decide whether to trust it. Per annotation cost fell sharply. Speaker info: Nachiket Paranjape: - https://x.com/nmparanjape - https://www.linkedin.com/in/nachiketparanjape/ Timestamps: 0:00 - The GenAI platform team, and its three forces 2:05 - Why eval became the fourth pillar 3:05 - UI first, then API first, then workflow first 4:01 - Evals as a team sport, not an engineering harness 4:57 - Who owns which part of quality 5:53 - The continuous loop: trace, sample, annotate, calibrate 7:42 - Telemetry and workflow as two surfaces 9:32 - Operators vibe coding their own annotation UIs 11:21 - Calibrating judge prompts, self serve 13:10 - Different teams, different prompt owners 14:04 - What it did to annotation cost --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/bMjlRrWjdT0-youtube-de0e357f07f3800aaed9 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/watch/bMjlRrWjdT0 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.