A talk by Nachiket Paranjape and Swaroop Chitlur Haridas (DoorDash) on running AI evaluations as a cross-functional practice rather than an engineering-only harness. Topics include: how DoorDash's GenAI platform team went API-first so non-engineering staff (strategy, operations) can build their own annotation tools via coding agents; the ownership structure where strategy/ops set the quality bar, PMs write rubrics, operations run annotations, and engineering supplies telemetry, datasets, and judges; the continuous eval loop of tracing, sampling, annotating, promoting golden sets, and calibrating judge prompts; self-serve judge calibration UIs; and how this approach reduced per-annotation cost. # Identity Observations — DoorDash GenAI Platform Team Talk ## Explicit Self-Description Evidence ### Speaker & Team Identity - The speakers introduce themselves as members of the **DoorDash GenAI Platform team**: *"we are here on behalf of the Door Dash GenAI platform team."* - The team describes itself as a **horizontal platform team**: *"We are a horizontal team that helps all other product teams... product teams at Door Dash build on top of the infrastructure and the primitives that we provide."* ### Claimed Value Proposition - The team states its **USP** explicitly: *"We see our USP and the value that we provide is that we help product teams balance these three forces which is accuracy, latency and cost."* ### Leadership Attribution - The team cites co-founder guidance as design influence: *"this was the guidance we had from Andy Fang our co-founder as well"* — regarding being UI-first. ## Claimed Scope ### Platform Pillars Described The speakers enumerate the platform's components: - **LLM gateway**: *"an LLM gateway where you can easily switch between different models"* - **Agent gateway**: *"an agent gateway where you can connect to tools and other agents"* - **Open-weights model hosting**: *"we invested in open weights models and have seen significant impact already"* - **Eval platform**: framed as *"the fourth pillar"* and the talk's focus ## Intended Audience ### Explicit Audience Markers - **Conference attendees**, post-lunch slot, with awareness of competing events: *"I know France is playing right now"* — indicating a live, in-person technical conference. - **Cross-references to other talks at the same event**: *"for those who attended Raga's talk earlier today," "I think we had a talk from Raghav before"* — indicating a shared conference audience. - **Practitioners familiar with AI tooling**: assumes knowledge of terms like **MCP, SDK, LLM-as-a-judge, trajectory-based evals, Codex, Claude Code, DSPy** without definition. ### Implicit Audience (from content) - **Engineers and platform builders** at other companies facing similar scaling problems. - **Non-engineer stakeholders** are addressed as beneficiaries rather than audience — the talk describes serving *"strategy and operations folks," "product managers," "labeling partners."* ## Observable Presentation Patterns ### Demonstrated Tone & Style - **Casual, conversational delivery** with humor: *"I promise you this will be better than that. I'm kidding."* - **Modest framing of the talk as shared learning**, not product marketing: *"we kind of wanted to share our eval journey," "we are all learning."* ### Demonstrated Themes - **Cross-functional empowerment as a core message**: repeated emphasis that evals are *"a team sport"* and *"not just an engineering harness."* - **Self-serve philosophy**: *"build something which is as self-served as possible so that people aren't always necessarily blocked by our team."* - **Evolution narrative**: the platform's posture progressed from *UI-first → API-first → workflow-first* across the talk. - **Quantified outcomes (creator-reported)**: *"we actually did see a lot of reduction in the spend at per annotation cost"* and *"resulted in faster loops"* — presented as results, not independently verified. ### Presentation Format - **Two-speaker, slide-driven conference talk** with an explicit time awareness (*"we have 4 minutes left"*) and Q&A offer (*"happy to hang out after the talk"*), including screenshots of the actual platform UI (*"the screenshot that you actually see is what actually exists"*). ## Claim vs. Demonstration Distinction | Category | Type | Notes | |---|---|---| | "USP is balancing accuracy, latency, cost" | **Creator claim** | Stated positioning, not demonstrated | | "Significant impact" from open-weights models | **Creator claim** | Deferred to a *"future conference"* | | Reduced annotation spend | **Creator-reported result** | No figures shown in transcript | | Platform architecture (gateways, telemetry, calibration UI) | **Demonstrated via detailed walkthrough and screenshots** | Concrete feature descriptions and workflow steps | | Cross-functional usage (StratOps/PMs vibecoding annotation UIs) | **Demonstrated via specific examples** | E.g., vibe-coded restaurant-menu annotation UI | # Delivery Observations: AI Evals for Cross-Functional Teams (DoorDash) ## Overall Format - **Two-speaker tag-team talk** with an explicit handoff: *"So with that context I'll hand it off to Nachig to talk about uh how we went about delivering this."* - The second speaker later interjects mid-flow, and the first briefly resumes for a closing comment (">> yeah just want to add to that is this enables different configurations..."), showing a lightly structured, conversational collaboration rather than rigidly scripted segments. ## Tone - **Casual, self-deprecating humor** used to warm up the audience: > "I know France is playing right now and I promise you this will be better than that. I'm kidding." - **Friendly, appreciative framing** at open and close: > "Good afternoon everyone. Thanks for uh coming for a post lunch uh talk. Always appreciate that." - Constant hedging/softening with *"you know," "kind of," "sort of," "uh," "um"* — a highly conversational register throughout. ## Vocabulary & Recurring Phrases - **"API first" / "UI first" / "workflow first"** — repeated framing of the platform's evolution: *"we kind of started with like okay we have to be UI first"* ... *"we kind of evolved to also being API first"* ... *"now we have become workflow first."* - **"Golden data sets"** repeated multiple times as a core term: *"then creating those golden data sets which are going to be you you know your uh golden data sets."* - **"Rinse and repeat" / "continuous loop"** to describe iteration: *"monitoring this over a period of time and then you know rinse and repeat uh go through the whole loop again."* - **"LLM as a judge" / "calibrate your judges"** repeated across both speakers. - **"Vibe coded" / "v coding"** used informally for partner-built UIs: *"we can actually enable our our partners to simply v code these UIs for annotation"* and *"a vibe coded UI looks pretty clean does the job."* - Informal modern vocabulary: *"latest and greatest," "the cool things," "nothing crazy."* ## Teaching Habits & Structure - **Numbered-step framing** — content is organized as sequential steps, restated at the end: > "finally I just wanted to you know quickly touch on this slide again uh the eight steps you know continuous loop" - **Audience callbacks** to earlier conference talks: *"for those who attended Ragago talk earlier today"* and *"I think we had a talk from Ragav before"* — situating the material in the event context. - **Concrete role enumeration** to explain cross-functional work: *"you're going to have your strategy and operations folks... you're going to have your product people... you're going to have your operations teams... you're going to have your engineering teams."* - **Analogy to everyday objects** to lower abstraction: *"this is basically like a menu from a restaurant uh it's it's you know nothing crazy."* - **Repeated summary/recap behavior** — both speakers restate prior points ("Uh and again going back uh and you know like just refreshing your memory. Uh step one capture your traces..."). - **Motivation-before-feature pattern**: features are introduced via the pain point first — *"what we wanted to do was really reduce the friction of back and forth with an engineering team."* ## Interaction Patterns - The second speaker **adds contextual caveats** to the first's claims (ownership varies by team: *"in some teams you have seen the strategy and operations folks own the prompt..."*), showing complementary rather than redundant roles. - The first speaker **affirms and extends**: *"Yeah, that that's a good point. I think the overall idea was to you know build something which is as self-s served as possible."* ## Uncertainty Notes - Transcript timestamps are coarse (second-level offsets, not reliable quote anchors), so timing patterns (e.g., pacing) cannot be observed. - Speaker attribution is inconsistent in the transcript (the opener introduces himself as "Farup"/"Harup"/"Surup" in different spots despite the title listing different names); the two main voices are distinguishable but the exact name spellings are uncertain. - Filler-word frequency ("uh," "um") is clearly high, but distinguishing nervousness from habitual style is not observable from text alone. # Observation Document: AI Evals for Cross-Functional Teams (DoorDash GenAI Platform Team) ## Speaker Identification - **Speaker 1 (Harup / "Surup" per transcript)** — introduces the DoorDash GenAI platform team context. (Note: transcript identifies him as "Farup"/"Harup"; curated metadata lists **Swaroop Chitlur Haridas**.) - **Speaker 2 (Nachiket)** — delivers the platform delivery details. - No curated speaker-role mapping is stated in the source; attribution below follows transcript self-identifications. ## Positions & Claims ### Evals as a Cross-Functional Effort - Both speakers frame the talk's central claim: evals began as "another engineering thing" but **evolved into a cross-functional effort** (Speaker 1; echoed by Speaker 2). - Speaker 2: "Evals is not just an engineering harness… from your traces to your data sets… this is all basically a team sport." ### Team Structure & Value Proposition (Speaker 1) - The GenAI platform team is **horizontal**, providing infrastructure/primitives for product teams at DoorDash. - Claimed value: helping teams balance **accuracy, latency, and cost** — applied initially to models, now also to agents. - Building blocks cited: **LLM gateway** (model switching), **agent gateway** (tool/agent connectivity, authentication, agent identity — "security team can bless"), **open-weights model hosting** (cost-driven; "significant impact already," details deferred to a future conference), and **eval** (today's topic). ### Diverse Team Needs (Speaker 1) - Different product teams had distinct eval needs: **session-level quality judgments** (consumer discovery/shopping assistant), **scaling human judgment** (personalization ML), **trajectory-based evals** (multi-agent systems). - Guiding decisions attributed to inputs including co-founder **Andy Fang**: be **UI-first** so non-engineers (strategy & ops, PMs, labeling partners) can contribute; then **API-first** for engineers; now **workflow-first** with coding agents letting non-engineers run operations. ### Continuous Quality Loop (Speaker 2) - Stated iteration cycle: **trace → sample → annotate → review → create golden data sets → monitor → repeat**. - Two platform surfaces: - **Telemetry layer**: traces, scores, observations; accessible via MCP, SDK, APIs. - **Workflow layer**: where strat-ops/PMs set annotation tasks, review golden data sets, create and calibrate judges. ### Annotation via Vibe-Coded UIs (Speaker 2) - Platform team couldn't build a UI per use case; instead, because of the **API-first** approach, strat-ops teams used **Codex or Claude Code** to build their own annotation UIs ("vibe coded UI"). - Examples cited: image annotation, manual testing use cases (referencing an earlier talk by "Ragav"). - Claimed insight: **underlying patterns were similar** across use cases. ### Judge Calibration (Speaker 2) - Process: start with a judge prompt → baseline scores on traces → optimization loop using the **DSPy library** (transcript: "JPEA") → partner teams elevate the optimized prompt as their LLM-as-judge. - Built a **self-serve UI** so PMs/operators run calibration themselves with any model (e.g., Gemini, Claude, OpenAI), removing engineering back-and-forth. - Emphasis on **reviewability**: showing original vs. calibrated system prompts to build partner trust; one example showed "a significant amount of improvement." ### Ownership Flexibility (Speaker 1, interjection) - The self-serve design enables varied team configurations: prompt owned by **strat-ops, PM, or engineering** depending on the team — "we are all learning… even the org design is improving and we are enabling that." ### Results (Speaker 2) - **Reduced per-annotation spend**; thousands of rows annotated weekly; self-serve platform increased velocity and reduced cost. - **Faster iteration loops**; teams calibrate judges independently, resulting in "very very high velocity." - Qualification: results described impressionistically ("really good results"); no quantitative figures given. ## Unresolved Tensions / Qualifications - Speaker 2 notes LLM-as-judge calibration "might be straightforward to a lot of folks but it is still like a pretty new and evolving field." - Open-weights cost impact claimed but explicitly deferred ("maybe we'll talk about that in a future conference"). - No metrics substantiate the claimed cost reduction or quality improvements. # Speaker Attribution & Provenance Record **Video:** AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash **Video ID:** `01a15ca7-9e36-7a77-b1c2-0f4ea5eb75fc` ## Speakers & Roles | Speaker | Role | Evidence | |---|---|---| | **First speaker ("Farup" / "Harup" / "Faroo" / "Surup", transcript spellings vary) | Co-presenter, DoorDash GenAI platform team | Self-introduction: *"my name is Farup and here's my teammate Nachiket. Uh we are here behalf of the Door Dash Genai platform team."* Title metadata suggests **Swaroop Chitlur Haridas**. | | **Second speaker ("Nachiket" / "Nachig")** | Co-presenter, DoorDash GenAI platform team | Handoff: *"I'll hand it off to Nachig to talk about how we went about delivering this."* Title metadata suggests **Nachiket Paranjape**. | - Both speakers alternate; the second speaker delivers the majority of the technical walkthrough, with brief interjections from the first (e.g., the `>>` exchange on prompt ownership across teams). - No narrator, guests, or other identified speaking roles. - Audience is non-speaking; **[applause]** recorded at end. ## Referenced Third Parties & Sources - **Andy Fang** — DoorDash co-founder; cited as source of "UI first" guidance (claim by speaker, not independently verified in transcript). - **Ragago / Ragav** (transcript spellings vary) — referenced as a prior talk at the same event on session-level quality judgments; speaker cites it as context only. - **JPEA library** (likely a transcription of "DSPy," uncertain) — cited as the prompt-optimization library used; name as transcribed. - **Models referenced:** Gemini, Claude, OpenAI models — mentioned as options in the calibration UI. - **Coding agents referenced:** "codeex" (Codex?), "claw code" (Claude Code?) — spellings per transcript, uncertain. ## Internal Roles Cited (non-speaking) - Strategy and operations folks, product managers, labeling/annotation partners, engineering teams, security team — described as participants in the eval workflow. ## Promotional Segments - None identified. No sponsor, product marketing, or call-to-action segments present. The talk is a conference presentation about DoorDash's internal platform. ## Comments Availability - **Not indicated** — no transcript or metadata regarding comment availability. ## Uncertainty Notes - First speaker's name appears in multiple inconsistent transcribed forms; matched to title metadata with low-moderate confidence. - Library and tool names are phonetic transcriptions and may be misrendered. Good afternoon everyone. Thanks for uh coming for a post lunch uh talk. Always appreciate that. Um my name is Farup and here's my teammate Nachiket. Uh we are uh here behalf of the Door Dash Genai platform team. Um and we kind of wanted to share our eval journey. Uh it started as uh uh you know eval is another engineering thing but then it slowly we realized it evolved into a cross functional effort and we kind of want to share our story here. So what is this team? This team is a gen platform team. Uh we are a horizontal team that helps all other product teams. So product teams at Door Dash build on top of the infrastructure and the primitives that we provide. Um and we see our uh USP and the value that we provide is that we help product teams balance these three forces which is accuracy, latency and cost. Um initially we applied this in terms of models but if you think about it it also applies to agents. Um and the way we achieve this is we have uh primitives and building blocks. Um so for example we have an LLM gateway where you can easily switch between different models uh and try the latest and greatest. Uh we have an agent gateway where you can connect to tools uh and other agents uh and we help solve authentication uh agent identity and other things in a central place uh which our security team can bless. Um similarly we pair the LLM gateway with open weights models hosting. Uh, of course cost is a number one concern these days. Uh, and we uh kind of invested in open weights models uh and have seen significant impact uh already. Um, and maybe we'll talk about that in a future conference. Uh, the fourth pillar is eval and that's the part that we would want to share today. Um when we started talking to product teams internally at Door Dash uh there were varying uh distinct needs across teams. We had a consumer discovery and shopping assistant team. Uh for those who attended Ragago talk earlier today uh you will uh see the need for session level quality judgments. um uh then the personalization ML then you needed uh a way to scale up human judgment and with multi- aent systems we needed trajectory based evals now the question is how do you cater to all these different needs under a common platform and uh as we spoke to these teams we realized like um we needed to empower the people who are the domain experts and in our case that was strategy and operations folks it as product managers uh it was even labeling partners uh and not only engineers so we kind of started with like okay we have to be UI first and this was the guidance we had from Andy Fang our co-founder as well um so we had UIs for non-engineers to contribute uh then we kind of evolved to also being API first so that engineers can also build and not be blocked on the central platform and they can build their own uh uh uh systems uh and Then of course with the coding agents now we have become workflow first where we kind of empower SNO and PMs to also being able to uh navigate uh the platform and uh run operations as well. Um so with that context I'll hand it off to Nachig to talk about uh how we went about delivering this. Cool. Thanks Harup. Um and thanks everyone for joining us. I know France is playing right now and I promise you this will be better than that. I'm kidding. Um so as Faroo was saying uh Evals is not just an engineering harness it is a cross functional effort across different pillars across different uh teams uh that actually helps us add all the domain specific knowledge into our uh into the quality of the AI itself. So from your traces to your data sets uh from you know scoring mechanisms uh this is all basically a team sport. we all have to play uh and help improve the quality of AI. So going a little bit deeper into the same aspect uh we have different uh teams uh at Door Dash who help us actually improve the quality of AI. So you're going to have your strategy and operations folks who are going to set priorities, set the quality bar that you want to aim for. You're going to have your product people who are going to translate uh these requirements into rubrics workflows. You're going to have your operations teams running uh annotations. You're going to have your engineering teams like us uh providing APIs, telemetry, data sets, judges, all you know the the cool things. Um and combining all these together is is what a recipe is for actually making sure that you are shipping quality AI products through an eval platform. So we've tried to boil this down uh into sort of you know like a a continuous iteration loop. Uh so right from tracing uh you know having a tracing solution viewing your sessions your traces to sampling them down you uh you know to a very small uh set that you actually want to look at uh annotating these with the domain specific expertise that you bring in with the different teams I mentioned reviewing those uh then creating those golden data sets which are going to be you you know your uh golden data sets and that that you want to measure or calibrate against uh and then of course like you know monitoring this over a period of time and then you know rinse and repeat uh go through the whole loop again. So this is in our experience has been you know like a good sort of continuous loop uh for you know shipping quality AI at the plat on on the platform level uh we we have two surfaces uh so we have the telemetry layer uh where we have all our traces our scores uh observations that is also sort of the plane where users are able to access these traces using an MCP using an SDK uh using our APIs and then we have the workflow This is where a lot of our strat ops, our product teams operate on the platform. So this is where all the annotation tasks are set. Uh you know this is where they review their golden data sets, uh create their judges, calibrate their judges and so on. So maybe today we'll go through you know these sort of four different uh modules or pillars of our platform uh step by step. Uh so again first one uh tracing and sampling uh which is actually capturing what your agents what your LLMs are actually uh you know outputting for the lack of better words uh and actually viewing those. Now in order to also power this uh whole platform we have I think as far mentioned we have gone in an API first uh approach. Uh what that has allowed us to do is have these table APIs that actually uh you know and then you know build UIs uh on top of that. Uh so all our scores our data sets uh these are all powered by very stable APIs uh that our team owns. Uh so all your API access uh including you know like an SDK access is basically powered by this single uh plane. Um again going back uh and you know like just refreshing your memory. Uh step one capture your traces uh capture your sessions uh measure your scores. Uh then you want to start uh almost you know like adding all your judgment your context your domain knowledge uh and then calibrating your judges is what we have seen as the whole uh life cycle. Step two is on the annotation side. Uh so you obviously are capturing a lot of your uh agentic behavior, your sessions, your traces, but you actually want to see what are some places where things went well and what are some places where things did not go well. This is where you can actually titrate your your you know and actually look in inside what's actually happening uh at the session level and annotate these data sets. Um and as Surup mentioned, we have a lot of use cases. we have we we talked to multiple different teams who have uh various uh ways of annotating uh their data sets uh and it's it's almost hard for a platform team to you know build like a UI specific uh for each use case uh and and you know to give you an example uh it's usually going to be an annotator who's going to annotate these data sets so the platform team is you know in charge of the APIs we have a strategy and of person who's actually deciding what to annotate and then you have an annotator who's actually going to annotate uh your data set. So we took this approach uh everybody uh has uh you know access to coding agents uh and we actually doubled down on that API first approach. So because we had these APIs we were actually uh able to enable our statops teams to use something like a codeex or a claw code and v code their own annotation UIs. Uh so we had different use cases. Uh I think we had a talk from Ragav before. Uh we had image annotation use cases. We had some uh you know manual testing use cases. What stood out to us was the underlying patterns were similar. So if we are are API first uh we can actually enable our our our partners to simply v code these UIs for annotation. So it's it's like a very simple example then you know of of a vibe coded UI looks pretty clean does the job uh and you get you know the annotation that you eat this is basically like a menu from a restaurant uh it's it's you know nothing crazy uh but the point I want to make here is that what helped us was to give this workflow in the hands of the operators so that they can actually build their own vcoded annotation UIs. Uh so moving on once you have these annotation UIs you obviously want to you know calibrate your your your judge prompts you obviously have some LM as a judge uh metric that you're tracking you want to now start improving that with these golden data sets u in order to do that uh you know we have a pretty simple process uh you you're going to start with you know some judge prompt take a look at you know what exactly do you want to measure from the output uh g you know have have us have something simple you're going to have your baseline scores uh where you're going to simply run those LLM judges on your traces and then you're going to have that optimization loop. Uh so we use uh the JPEA library which is a pretty commonly used library out there for prompt optimization. Uh and once you know the the iteration loop is complete uh our partner teams are happy they're going to then elevate that judge prompt as their LLM as a judge. Now even while doing that uh LLM as a judge as a concept the whole prompt calibration concept might be uh straightforward to a lot of folks but it is still like a pretty new and evolving field. Uh and what we wanted to do was really reduce the friction of back and forth with an engineering team. So we tried to really remove all the complicated logic and make this into a self-s serve UI. So the screenshot that you actually see is what actually exists. uh so uh you know like a product manager or an operator is going to come to our UI. They're going to set some of these configs uh on the platform and then actually run the calibration loop themselves. So they don't have to worry about the different settings that they need to worry about what are the different uh you know tweaks that they need to do and they can actually like you know run a calibration loop using any model of their choice. I think in this example I have Gemini they can use run it using uh you know any of the claude or the openi models too. The other important piece was actually uh making this reviewable. Uh you know again a lot of this uh is a closed box where you can't really it's hard to see what's actually happening. Uh so the second piece that we built was actually giving them vis visualization and visibility into what's actually happening. So on the left you can see we and this is like one of the good examples where we saw like a significant amount of improvement in the judge prompt. uh and we actually show the you know the the previous the original system prompt and the calibrated prompt to our partners so that they are also able to gain that trust uh why as as as we build this \>\> yeah just want to add to that is this enables different configurations in different teams in some teams you have seen the strategy and operations folks own the prompt uh you have seen some teams where the product manager owns the prompt you have seen some teams where engineering owns the prompt so this gives gives the flexibility for teams to design and evolve because we are all learning. So the even the org uh design is improving and we are enabling that. \>\> Yeah, that that's a good point. I think the overall idea was to you know build something which is as self-s served as possible so that uh you know people aren't always necessarily blocked by our team helping them out. Um and then finally you know uh the quality loop in practice. you know as we've been going through this exercise we've seen a lot of improvements happening to our product as well. So you know for example we sort mentioned we started with the UIs we are you know now API and workflow first uh we're trying to reuse a lot of the existing infrastructure that already existed at Door Dash uh and that's helped us uh get a long way. Now some of uh we we we've seen obviously like you know really good results. I think a very good result that we we do like to call out is we actually did see a lot of reduction in the spend uh at per annotation cost as you as you all can imagine we do have you know thousands of rows that need to get annotated every week uh and it can get pretty expensive at doash scale uh and having this selfserve uh annotation platform really helped us reduce increase the velocity and reduce the cost that we were actually spending with these annotators. to to annotate the data for us. Uh obviously uh this resulted in faster loops. Uh teams were able to iterate faster. They were able to uh you know calibrate their own judges in a completely self-s served way. Uh and thus it has resulted us in in in moving with a very very high velocity. So uh finally I just wanted to you know quickly touch on this slide again uh the eight steps you know continuous loop uh which is you know you you have your traces you want to look at your traces your sessions you want to sample it down to a size which is which you are comfortable with uh you want to start annotating your data sets you really want to start uh making the data better with the human knowledge that exists and the domain knowledge that exists and then calibrate your workflows was calibrate your agents, calibrate your LLM judges with this golden data set and then repeat this whole cycle uh over a period of time to you know to ship reliably and ship with high quality. Um yeah, we have 4 minutes left. Thank you once again. I think that was the last slide. Uh thanks for attending and if there's any questions, we'd be happy to hang out after the talk or even happy to answer them now. \>\> \[applause\]