--- schema: 1 type: video title: >- Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust description: >- Same accuracy. Four times the cost. What happened when Braintrust pitted vector search against agentic search for a coding agent. Jess Wang, developer advocate at Braintrust, starts with the basics of evals: why shipping on vibes fails, the four parts of an eval (dataset, task, scorer, experiments), and how observability and evals form a flywheel. Then she runs a real one. Using fix PRs from Microsoft's TypeScript Go repo, she has Claude Code find the buggy code with either agentic search (grep, find, reading files) or vector search over embeddings, scored by the repo's own test suite. Both reached the same accuracy, but vector search cost about four times as much because chunks lacked the surrounding context and the agent kept searching. In this talk: • What evals are made of, and why they're a team sport • Building an eval dataset from real merged fix PRs • Forcing Claude Code into vector-only search, and fixing orphaned subprocess traces • Why vector search found nearby code but missed the connective tissue SPEAKER Jess Wang, Developer Advocate, Braintrust LinkedIn: https://www.linkedin.com/in/jessica-wang-a1aa15ba/ X: https://x.com/daRubberDuckiee YouTube: https://www.youtube.com/@theRubberDuckiee LINKS Braintrust: https://www.braintrust.dev/ CHAPTERS 0:00 Intro 0:42 What evals are 1:02 Don't ship on vibes 1:27 What evals answer 2:22 The sycophancy rollback 3:02 The four parts of an eval 4:32 Comparing experiments 4:52 Querying traces in plain English 5:47 Observability 6:02 The eval flywheel 7:47 Evals are a team sport 8:42 The experiment: agentic vs vector search 9:32 How vector search works 10:42 How agentic search works 11:22 Building the dataset 12:17 Running both in Claude Code 13:17 Fixing orphaned traces 14:37 Scoring with the test suite 14:57 Results: same accuracy, 4x the cost 15:22 Why vector search fell short 16:27 Why it cost more 16:47 Caveats and best practices Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMEvals #CodingAgents #AIEngineer canonical: 'https://liquidgold.cc/watch/T3SS931wU0I' markdown: 'https://liquidgold.cc/watch/T3SS931wU0I.md' llms: 'https://liquidgold.cc/watch/T3SS931wU0I/llms.txt' parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:19:57.784Z' language: null children: [] videoId: T3SS931wU0I durationSeconds: 1095 publishedAt: '2026-10-05T15:30:36.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 24 endSeconds: 45 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=24' heldReason: rights - ordinal: 1 text: null startSeconds: 56 endSeconds: 68 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=56' heldReason: rights - ordinal: 2 text: null startSeconds: 90 endSeconds: 101 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=90' heldReason: rights - ordinal: 3 text: null startSeconds: 145 endSeconds: 156 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=145' heldReason: rights - ordinal: 4 text: null startSeconds: 191 endSeconds: 203 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=191' heldReason: rights - ordinal: 5 text: null startSeconds: 203 endSeconds: 276 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=203' heldReason: rights - ordinal: 6 text: null startSeconds: 276 endSeconds: 289 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=276' heldReason: rights - ordinal: 7 text: null startSeconds: 301 endSeconds: 311 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=301' heldReason: rights - ordinal: 8 text: null startSeconds: 356 endSeconds: 366 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=356' heldReason: rights - ordinal: 9 text: null startSeconds: 366 endSeconds: 378 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=366' heldReason: rights - ordinal: 10 text: null startSeconds: 467 endSeconds: 477 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=467' heldReason: rights - ordinal: 11 text: null startSeconds: 545 endSeconds: 581 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=545' heldReason: rights - ordinal: 12 text: null startSeconds: 593 endSeconds: 647 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=593' heldReason: rights - ordinal: 13 text: null startSeconds: 647 endSeconds: 658 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=647' heldReason: rights - ordinal: 14 text: null startSeconds: 692 endSeconds: 705 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=692' heldReason: rights - ordinal: 15 text: null startSeconds: 739 endSeconds: 749 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=739' heldReason: rights - ordinal: 16 text: null startSeconds: 805 endSeconds: 816 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=805' heldReason: rights - ordinal: 17 text: null startSeconds: 894 endSeconds: 904 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=894' heldReason: rights - ordinal: 18 text: null startSeconds: 904 endSeconds: 915 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=904' heldReason: rights - ordinal: 19 text: null startSeconds: 927 endSeconds: 938 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=927' heldReason: rights - ordinal: 20 text: null startSeconds: 1019 endSeconds: 1095 watch: 'https://liquidgold.cc/watch/T3SS931wU0I?t=1019' heldReason: rights --- # Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust Same accuracy. Four times the cost. What happened when Braintrust pitted vector search against agentic search for a coding agent. Jess Wang, developer advocate at Braintrust, starts with the basics of evals: why shipping on vibes fails, the four parts of an eval (dataset, task, scorer, experiments), and how observability and evals form a flywheel. Then she runs a real one. Using fix PRs from Microsoft's TypeScript Go repo, she has Claude Code find the buggy code with either agentic search (grep, find, reading files) or vector search over embeddings, scored by the repo's own test suite. Both reached the same accuracy, but vector search cost about four times as much because chunks lacked the surrounding context and the agent kept searching. In this talk: • What evals are made of, and why they're a team sport • Building an eval dataset from real merged fix PRs • Forcing Claude Code into vector-only search, and fixing orphaned subprocess traces • Why vector search found nearby code but missed the connective tissue SPEAKER Jess Wang, Developer Advocate, Braintrust LinkedIn: https://www.linkedin.com/in/jessica-wang-a1aa15ba/ X: https://x.com/daRubberDuckiee YouTube: https://www.youtube.com/@theRubberDuckiee LINKS Braintrust: https://www.braintrust.dev/ CHAPTERS 0:00 Intro 0:42 What evals are 1:02 Don't ship on vibes 1:27 What evals answer 2:22 The sycophancy rollback 3:02 The four parts of an eval 4:32 Comparing experiments 4:52 Querying traces in plain English 5:47 Observability 6:02 The eval flywheel 7:47 Evals are a team sport 8:42 The experiment: agentic vs vector search 9:32 How vector search works 10:42 How agentic search works 11:22 Building the dataset 12:17 Running both in Claude Code 13:17 Fixing orphaned traces 14:37 Scoring with the test suite 14:57 Results: same accuracy, 4x the cost 15:22 Why vector search fell short 16:27 Why it cost more 16:47 Caveats and best practices Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMEvals #CodingAgents #AIEngineer Canonical: https://liquidgold.cc/watch/T3SS931wU0I · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks --- #### Chunk 0: Document excerpt (0:12 to 1:30) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 1: Document excerpt (1:30 to 3:11) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 2: Document excerpt (3:11 to 4:36) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 3: Document excerpt (4:36 to 6:06) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 4: Document excerpt (6:06 to 7:47) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 5: Document excerpt (7:47 to 8:54) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 6: Document excerpt (8:54 to 10:47) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 7: Document excerpt (10:47 to 12:19) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 8: Document excerpt (12:19 to 13:36) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 9: Document excerpt (13:36 to 15:15) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 10: Document excerpt (15:15 to 16:48) > Held: transcript rights do not permit this chunk's text. --- #### Chunk 11: Document excerpt (16:48 to 18:15) > Held: transcript rights do not permit this chunk's text.