--- schema: 1 type: video title: 'Evals in AI: A Deep Dive — Tejas Kumar, IBM' description: >- A customer support answer passes a test because it contains the word “cannot,” even though it approves a return outside the store's policy. Tejas Kumar builds that failure live to show where ordinary assertions and fuzzy matching stop being useful. The example grows into an LLM judge, then a dataset of customer scenarios, agent responses, and human verdicts. Along the way, the judge starts favoring an answer generated by its own model family. Adding the actual return policy improves its decisions, but missing details and a loyal customer's appeal still expose gaps. Kumar keeps examining disagreements, supplying context, and eventually changing the model rather than treating a green test as proof that the system works. The workshop connects those experiments to an evaluation process that can survive production. Test cases, expected behavior, a judge, and aggregation form the basic structure. Position bias, sycophancy, self preference, and verbosity can distort the result, so the instrument itself needs calibration. Kumar sets an 80% agreement threshold for a CI gate, then explains why fresh traffic and human review must keep the dataset current. An OpenRAG demonstration retrieves an uploaded refund policy instead of relying on a rule buried in a prompt. He also distinguishes checks performed before deployment from harness protections at runtime, and recommends starting with deterministic checks before paying for model judgments. Audience questions explore synthetic examples, shared policy, and where evaluation belongs in the development loop. Speaker info: - https://x.com/tejaskumar_ - https://linkedin.com/in/tejasq - https://tej.as/blog/ai-evals-llm-judge Timestamps: 0:00 - Introduction 2:03 - Why evals and harnesses work together 6:09 - Evals and unsafe agent behavior 9:47 - Fuzzy unit tests and evaluation components 15:21 - Evaluation techniques 18:30 - Four ways judges can mislead 21:11 - Calibrating a judge against human verdicts 23:30 - CI gates and keeping datasets current 26:15 - Retrieving living policy with OpenRAG 28:55 - Live coding from a unit test 31:26 - A misleading substring match 32:08 - Building and testing an LLM judge 36:19 - Calibrating against a scenario dataset 43:00 - Refining policy and changing the model 46:08 - Turning agreement into a CI gate 48:12 - A policy retrieval demo 51:53 - Recap and evaluation costs 54:36 - Audience questions --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/NdrSPm6NCdk-youtube-532f0bf69e9eae3a73fa Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/evals-in-ai-a-deep-dive-tejas-kumar-ibm-7edc162c32a4 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/evals-in-ai-a-deep-dive-tejas-kumar-ibm-7edc162c32a4.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:19:54.699Z' language: null children: [] videoId: NdrSPm6NCdk durationSeconds: 3589 publishedAt: '2026-10-05T17:00:22.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 13 endSeconds: 124 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=13' heldReason: rights - ordinal: 1 text: null startSeconds: 134 endSeconds: 369 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=134' heldReason: rights - ordinal: 2 text: null startSeconds: 134 endSeconds: 369 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=134' heldReason: rights - ordinal: 3 text: null startSeconds: 381 endSeconds: 597 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=381' heldReason: rights - ordinal: 4 text: null startSeconds: 608 endSeconds: 697 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=608' heldReason: rights - ordinal: 5 text: null startSeconds: 697 endSeconds: 707 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=697' heldReason: rights - ordinal: 6 text: null startSeconds: 707 endSeconds: 818 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=707' heldReason: rights - ordinal: 7 text: null startSeconds: 828 endSeconds: 921 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=828' heldReason: rights - ordinal: 8 text: null startSeconds: 932 endSeconds: 1116 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=932' heldReason: rights - ordinal: 9 text: null startSeconds: 932 endSeconds: 1116 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=932' heldReason: rights - ordinal: 10 text: null startSeconds: 1127 endSeconds: 1271 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=1127' heldReason: rights - ordinal: 11 text: null startSeconds: 1127 endSeconds: 1271 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=1127' heldReason: rights - ordinal: 12 text: null startSeconds: 1281 endSeconds: 1414 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=1281' heldReason: rights - ordinal: 13 text: null startSeconds: 1281 endSeconds: 1414 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=1281' heldReason: rights - ordinal: 14 text: null startSeconds: 1414 endSeconds: 1425 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=1414' heldReason: rights - ordinal: 15 text: null startSeconds: 1425 endSeconds: 1579 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=1425' heldReason: rights - ordinal: 16 text: null startSeconds: 1425 endSeconds: 1579 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=1425' heldReason: rights - ordinal: 17 text: null startSeconds: 1425 endSeconds: 1579 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=1425' heldReason: rights - ordinal: 18 text: null startSeconds: 1579 endSeconds: 1590 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=1579' heldReason: rights - ordinal: 19 text: null startSeconds: 1899 endSeconds: 1933 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=1899' heldReason: rights - ordinal: 20 text: null startSeconds: 1943 endSeconds: 2182 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=1943' heldReason: rights - ordinal: 21 text: null startSeconds: 2194 endSeconds: 2276 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=2194' heldReason: rights - ordinal: 22 text: null startSeconds: 2287 endSeconds: 2386 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=2287' heldReason: rights - ordinal: 23 text: null startSeconds: 2488 endSeconds: 2587 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=2488' heldReason: rights - ordinal: 24 text: null startSeconds: 2597 endSeconds: 2779 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=2597' heldReason: rights - ordinal: 25 text: null startSeconds: 2792 endSeconds: 2900 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=2792' heldReason: rights - ordinal: 26 text: null startSeconds: 2912 endSeconds: 3118 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=2912' heldReason: rights - ordinal: 27 text: null startSeconds: 3129 endSeconds: 3279 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=3129' heldReason: rights - ordinal: 28 text: null startSeconds: 3129 endSeconds: 3279 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=3129' heldReason: rights - ordinal: 29 text: null startSeconds: 3291 endSeconds: 3589 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=3291' heldReason: rights - ordinal: 30 text: null startSeconds: 3291 endSeconds: 3589 watch: 'https://liquidgold.cc/watch/NdrSPm6NCdk?t=3291' heldReason: rights --- # Evals in AI: A Deep Dive — Tejas Kumar, IBM A customer support answer passes a test because it contains the word “cannot,” even though it approves a return outside the store's policy. Tejas Kumar builds that failure live to show where ordinary assertions and fuzzy matching stop being useful. The example grows into an LLM judge, then a dataset of customer scenarios, agent responses, and human verdicts. Along the way, the judge starts favoring an answer generated by its own model family. Adding the actual return policy improves its decisions, but missing details and a loyal customer's appeal still expose gaps. Kumar keeps examining disagreements, supplying context, and eventually changing the model rather than treating a green test as proof that the system works. The workshop connects those experiments to an evaluation process that can survive production. Test cases, expected behavior, a judge, and aggregation form the basic structure. Position bias, sycophancy, self preference, and verbosity can distort the result, so the instrument itself needs calibration. Kumar sets an 80% agreement threshold for a CI gate, then explains why fresh traffic and human review must keep the dataset current. An OpenRAG demonstration retrieves an uploaded refund policy instead of relying on a rule buried in a prompt. He also distinguishes checks performed before deployment from harness protections at runtime, and recommends starting with deterministic checks before paying for model judgments. Audience questions explore synthetic examples, shared policy, and where evaluation belongs in the development loop. Speaker info: - https://x.com/tejaskumar\_ - https://linkedin.com/in/tejasq - https://tej.as/blog/ai-evals-llm-judge Timestamps: 0:00 - Introduction 2:03 - Why evals and harnesses work together 6:09 - Evals and unsafe agent behavior 9:47 - Fuzzy unit tests and evaluation components 15:21 - Evaluation techniques 18:30 - Four ways judges can mislead 21:11 - Calibrating a judge against human verdicts 23:30 - CI gates and keeping datasets current 26:15 - Retrieving living policy with OpenRAG 28:55 - Live coding from a unit test 31:26 - A misleading substring match 32:08 - Building and testing an LLM judge 36:19 - Calibrating against a scenario dataset 43:00 - Refining policy and changing the model 46:08 - Turning agreement into a CI gate 48:12 - A policy retrieval demo 51:53 - Recap and evaluation costs 54:36 - Audience questions --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/NdrSPm6NCdk-youtube-532f0bf69e9eae3a73fa Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/evals-in-ai-a-deep-dive-tejas-kumar-ibm-7edc162c32a4 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.