--- schema: 1 type: video title: >- Vector Isn't Enough: Hybrid Search & Retrieval — Jeff Vestal & James Williams, Elastic description: >- An agent diagnoses exit code 137 by retrieving documentation, explaining an out of memory failure, and pointing to settings that can prevent it. Its answer depends on the search results it receives, and its citations lead back to the underlying documents. Jeff Vestal and James Williams work backward from that agent behavior to the retrieval layer in Elasticsearch. The hands on labs compare semantic search with BM25, showing why similar meaning can miss an exact identifier and why exact keywords can miss a paraphrase. Participants run queries in the Dev Tools console and Python notebooks, inspect rankings, and combine both methods rather than assuming one search strategy fits every question. The workshop explains reciprocal rank fusion, which combines result positions, and linear combination, which normalizes scores and exposes weights for tuning. Filtering narrows the candidate pool, judgment sets help assess changes, and search templates offer a way to route different query types to different strategies. The Agent Builder demonstration connects retrieval to tools, skills, repeated searches, and answers with sources that viewers can inspect. A bonus section compares pointwise and listwise reranking and asks when improved precision justifies another inference step. Questions cover chunking, embedding consistency, quantization, latency, and the limits of automatic tuning. The through line is practical: improve and measure the evidence reaching an agent before expecting the model to rescue poor retrieval. Speaker info: - https://www.linkedin.com/in/jeffvestal - https://www.linkedin.com/in/james-williams-b509341 - https://github.com/jeffvestal/elastic-vectorsearch-workshop Timestamps: 0:00 - Jeff, James, and the workshop environment 2:04 - Retrieval still matters for agents 3:14 - Vector, lexical, and hybrid labs 6:35 - Inference inside Elasticsearch 7:43 - Chunking and semantic text 12:02 - The hybrid search thesis 14:24 - Embeddings and semantic retrieval 17:08 - Where exact identifiers break vector search 18:06 - Pagination and document summaries 19:56 - Quantization and vector efficiency 21:23 - Graph retrieval questions 26:07 - Reciprocal rank fusion controls 27:38 - Relevance and latency tradeoffs 29:17 - Lexical search strengths and limits 30:39 - Inference and end to end latency 33:52 - Weight tuning and future automation 37:00 - Combine rankings with RRF 39:40 - Filter the candidate pool 40:37 - Normalize scores for linear combination 41:05 - Judgment sets and experiments 41:52 - Route queries with search templates 44:13 - Retrieval quality and model answers 45:32 - Agent Builder and workflows 47:31 - Reuse agent APIs in an application 50:16 - Answers with verifiable sources 53:14 - Skills and search tools 55:23 - ES|QL and index search 57:57 - Diagnose exit code 137 59:21 - ES|QL for agentic retrieval 63:35 - Inspect results and citations 65:28 - Reranking as a second stage 68:18 - Decide when reranking is worth it 69:59 - Pointwise and listwise rerankers 72:08 - Run retrieval and reranking together 73:06 - Workshop resources and wrap up --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/ksjwtQop4os-youtube-44ce97714e055ecf5f66 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- canonical: >- https://liquidgold.cc/channel/aidotengineer/explore/video/vector-isn-t-enough-hybrid-search-retrieval-jeff-ve-f0e142d2fc52 markdown: >- https://liquidgold.cc/channel/aidotengineer/video/vector-isn-t-enough-hybrid-search-retrieval-jeff-ve-f0e142d2fc52.md llms: null parent: 'https://liquidgold.cc/channel/aidotengineer.md' channel: aidotengineer updated: '2026-10-23T22:19:18.157Z' language: null children: [] videoId: ksjwtQop4os durationSeconds: 4580 publishedAt: '2026-10-11T13:00:18.000Z' searchChunks: - ordinal: 0 text: null startSeconds: 144 endSeconds: 198 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=144' heldReason: rights - ordinal: 1 text: null startSeconds: 209 endSeconds: 400 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=209' heldReason: rights - ordinal: 2 text: null startSeconds: 209 endSeconds: 400 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=209' heldReason: rights - ordinal: 3 text: null startSeconds: 413 endSeconds: 467 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=413' heldReason: rights - ordinal: 4 text: null startSeconds: 478 endSeconds: 728 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=478' heldReason: rights - ordinal: 5 text: null startSeconds: 478 endSeconds: 728 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=478' heldReason: rights - ordinal: 6 text: null startSeconds: 740 endSeconds: 867 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=740' heldReason: rights - ordinal: 7 text: null startSeconds: 877 endSeconds: 1038 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=877' heldReason: rights - ordinal: 8 text: null startSeconds: 877 endSeconds: 1038 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=877' heldReason: rights - ordinal: 9 text: null startSeconds: 1050 endSeconds: 1086 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=1050' heldReason: rights - ordinal: 10 text: null startSeconds: 1102 endSeconds: 1203 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=1102' heldReason: rights - ordinal: 11 text: null startSeconds: 1214 endSeconds: 1288 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=1214' heldReason: rights - ordinal: 12 text: null startSeconds: 1299 endSeconds: 1567 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=1299' heldReason: rights - ordinal: 13 text: null startSeconds: 1299 endSeconds: 1567 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=1299' heldReason: rights - ordinal: 14 text: null startSeconds: 1578 endSeconds: 1658 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=1578' heldReason: rights - ordinal: 15 text: null startSeconds: 1672 endSeconds: 1761 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=1672' heldReason: rights - ordinal: 16 text: null startSeconds: 1771 endSeconds: 1843 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=1771' heldReason: rights - ordinal: 17 text: null startSeconds: 1854 endSeconds: 2037 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=1854' heldReason: rights - ordinal: 18 text: null startSeconds: 2048 endSeconds: 2225 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=2048' heldReason: rights - ordinal: 19 text: null startSeconds: 2237 endSeconds: 2382 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=2237' heldReason: rights - ordinal: 20 text: null startSeconds: 2395 endSeconds: 2443 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=2395' heldReason: rights - ordinal: 21 text: null startSeconds: 2453 endSeconds: 2465 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=2453' heldReason: rights - ordinal: 22 text: null startSeconds: 2476 endSeconds: 2522 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=2476' heldReason: rights - ordinal: 23 text: null startSeconds: 2532 endSeconds: 2653 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=2532' heldReason: rights - ordinal: 24 text: null startSeconds: 2742 endSeconds: 2851 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=2742' heldReason: rights - ordinal: 25 text: null startSeconds: 2864 endSeconds: 3021 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=2864' heldReason: rights - ordinal: 26 text: null startSeconds: 3033 endSeconds: 3202 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=3033' heldReason: rights - ordinal: 27 text: null startSeconds: 3338 endSeconds: 3480 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=3338' heldReason: rights - ordinal: 28 text: null startSeconds: 3491 endSeconds: 3569 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=3491' heldReason: rights - ordinal: 29 text: null startSeconds: 3580 endSeconds: 3818 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=3580' heldReason: rights - ordinal: 30 text: null startSeconds: 3947 endSeconds: 4098 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=3947' heldReason: rights - ordinal: 31 text: null startSeconds: 4108 endSeconds: 4199 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=4108' heldReason: rights - ordinal: 32 text: null startSeconds: 4209 endSeconds: 4328 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=4209' heldReason: rights - ordinal: 33 text: null startSeconds: 4338 endSeconds: 4386 watch: 'https://liquidgold.cc/watch/ksjwtQop4os?t=4338' heldReason: rights --- # Vector Isn't Enough: Hybrid Search & Retrieval — Jeff Vestal & James Williams, Elastic An agent diagnoses exit code 137 by retrieving documentation, explaining an out of memory failure, and pointing to settings that can prevent it. Its answer depends on the search results it receives, and its citations lead back to the underlying documents. Jeff Vestal and James Williams work backward from that agent behavior to the retrieval layer in Elasticsearch. The hands on labs compare semantic search with BM25, showing why similar meaning can miss an exact identifier and why exact keywords can miss a paraphrase. Participants run queries in the Dev Tools console and Python notebooks, inspect rankings, and combine both methods rather than assuming one search strategy fits every question. The workshop explains reciprocal rank fusion, which combines result positions, and linear combination, which normalizes scores and exposes weights for tuning. Filtering narrows the candidate pool, judgment sets help assess changes, and search templates offer a way to route different query types to different strategies. The Agent Builder demonstration connects retrieval to tools, skills, repeated searches, and answers with sources that viewers can inspect. A bonus section compares pointwise and listwise reranking and asks when improved precision justifies another inference step. Questions cover chunking, embedding consistency, quantization, latency, and the limits of automatic tuning. The through line is practical: improve and measure the evidence reaching an agent before expecting the model to rescue poor retrieval. Speaker info: - https://www.linkedin.com/in/jeffvestal - https://www.linkedin.com/in/james-williams-b509341 - https://github.com/jeffvestal/elastic-vectorsearch-workshop Timestamps: 0:00 - Jeff, James, and the workshop environment 2:04 - Retrieval still matters for agents 3:14 - Vector, lexical, and hybrid labs 6:35 - Inference inside Elasticsearch 7:43 - Chunking and semantic text 12:02 - The hybrid search thesis 14:24 - Embeddings and semantic retrieval 17:08 - Where exact identifiers break vector search 18:06 - Pagination and document summaries 19:56 - Quantization and vector efficiency 21:23 - Graph retrieval questions 26:07 - Reciprocal rank fusion controls 27:38 - Relevance and latency tradeoffs 29:17 - Lexical search strengths and limits 30:39 - Inference and end to end latency 33:52 - Weight tuning and future automation 37:00 - Combine rankings with RRF 39:40 - Filter the candidate pool 40:37 - Normalize scores for linear combination 41:05 - Judgment sets and experiments 41:52 - Route queries with search templates 44:13 - Retrieval quality and model answers 45:32 - Agent Builder and workflows 47:31 - Reuse agent APIs in an application 50:16 - Answers with verifiable sources 53:14 - Skills and search tools 55:23 - ES|QL and index search 57:57 - Diagnose exit code 137 59:21 - ES|QL for agentic retrieval 63:35 - Inspect results and citations 65:28 - Reranking as a second stage 68:18 - Decide when reranking is worth it 69:59 - Pointwise and listwise rerankers 72:08 - Run retrieval and reranking together 73:06 - Workshop resources and wrap up --- Watch along with the corrected transcript, summary, timestamps and resources on this talk's official AIE page: https://ai.engineer/talks/ksjwtQop4os-youtube-44ce97714e055ecf5f66 Join our upcoming AI Engineer conferences (see https://ai.engineer landing page): - AIEi Shanghai, November 5–6 — the first for meeting top Chinese labs and engineers - AIE CODE in SF, November 10–12 — the top AI Coding conference returns - AIEi Sydney, December 7–8 — the first AIEi run alongside NeurIPS 2026 in Sydney --- Canonical: https://liquidgold.cc/channel/aidotengineer/explore/video/vector-isn-t-enough-hybrid-search-retrieval-jeff-ve-f0e142d2fc52 · Channel: [AI Engineer](https://liquidgold.cc/channel/aidotengineer.md) > Search chunk text is held until public-source provenance is verified. ## Document chunks > Missing: no stored document chunks are available.