A talk by Sohail Shaikh and Ankush Rastogi (Prosodica) examining why "fat agent" architectures that load large tool catalogs into the system prompt cause latency, cost, and reliability problems. It covers the Semantic Tool Router pattern, a deterministic layer that reduces context shown to the model at inference time, with benchmarks across frontier models (GPT-4o, Gemini 2.0) measuring Time-to-First-Token latency and tool-selection accuracy as tool count grows. The session describes moving from static tool loading to Just-in-Time Context Injection—the speakers report up to 90% latency reduction in high-tool-density scenarios—and closes with a framework for building tool routers that scale to hundreds of capabilities. # Channel Identity Observations: Sohail Shaikh & Ankush Rastogi, Prosodica ## Video Presentation Overview This is a **technical talk/presentation format video** (~30 minutes by the speakers' own estimate) delivered by two presenters from the company **Prosodica** (spelled "Presodica" in the transcript's spoken rendering). The talk covers an AI engineering topic: why giving LLM agents large tool catalogs degrades performance, and how semantic routing fixes it. ## Explicit Self-Description Evidence ### Speaker Credentials (Creator Claims) - **Sohail Shaikh** identifies himself as: > "a data scientist with Presodica. My background spans across AI, NLP, marketing, analytics, and even engineering. My current focus is on applied AI, NLP, and conversational intelligence along with rack systems [RAG systems]." - **Ankush Rastogi** identifies himself as: > "a senior data solutions engineer at Presodica. I have spent more than a decade in AI, data engineering, and production systems. My focus is the engineering side." - Ankush explicitly frames the production angle: > "it's not about what's going to work in notebook, but whether it's going to survive in with real load, real user, and real failures." ### Channel/Company Association - Both speakers state they work at **Prosodica**, a company name attached to the talk in the title itself. No further claims about the company's business are made. ## Intended Audience (Demonstrated) The presentation targets a **technical builder audience**: - Developers building AI agents and RAG systems — evidenced by references to vector databases (Chroma DB, Pinecone, Qdrant), embedding models, cosine similarity, and JSON schemas. - Engineers concerned with **production concerns** — cost per thousand calls, time to first token, latency, logging, and monitoring. - Assumed familiarity with concepts like RAG: > "If you have built a rack system previously, this will feel pretty similar to that." Closing call-to-action confirms the audience targeting: > "please share it with someone building agents. They have either already hit this wall or they're about to." ## Claimed Scope of Content - **Topic scope**: A single, focused engineering problem — the "100-tool agent trap" — with claimed coverage of: - Why large tool catalogs break agent accuracy ("we'll show why it breaks, what the numbers look like") - A solution pattern ("semantic routing with just-in-time context") - Implementation guidance, benchmarks, trade-offs, and a production checklist - **Benchmark claims**: The speakers present specific results (e.g., fat agent accuracy "78% at 10 tools" falling to "13% at 1041 tools"; router "stays above 83%") from datasets including "Berkeley function calling leaderboard, Skills bench style scenarios, and synthetic tool pools." These are creator-reported figures, not independently verifiable within the video. - **Third-party references**: Anthropic's MCP on-demand tool loading write-up ("150k tokens down to 2,000"), Aurelio Labs' semantic router, ToolBench, and MCP Zero are cited as supporting ecosystem evidence. ## Demonstrated Presentation Patterns - **Two-presenter lecture format** with a clear division of labor, explicitly stated by Ankush: > "Suhail will focus more on the model and routing behavior, and I will focus more on system design, implementation, and production trade-offs." - **Slide-driven structure** — repeated references ("on this slide," "on the left side," "next slide") indicate a slide deck accompanies the talk. - **Practical, implementation-oriented tone** — includes a code walkthrough, a six-step production checklist, and pragmatic caveats ("If you have fewer than 20 tools, a router may be unnecessary"). - **Professional networking intent** — closing slide includes "LinkedIn QR codes on this slide to connect with us," indicating the talk doubles as professional/company visibility. ## Claims vs. Demonstrated Distinction - **Creator claims**: credentials, benchmark numbers, token-reduction figures, and the asserted generality of the solution ("For most team, it is a focused sprint"). - **Demonstrated**: deep applied-AI expertise in agent architecture, familiarity with the RAG/vector-search tooling ecosystem, and a production-engineering orientation — all consistently shown through the technical content itself rather than merely asserted. # Delivery Observations: "The 100-Tool Agent Is a Trap" ## Speaker Structure - **Two-presenter format** with an explicit division of labor, stated early by Ankush: *"Suhail will focus more on the model and routing behavior, and I will focus more on system design, implementation, and production trade-offs."* - **Speaker handoffs are verbal and polite**, often with thanks: *"Thanks, Ankush. So, let's now look into how semantic routing works."* and *"Thank you, Sohil. Now, we will discuss semantic routing in action."* - The transcript shows **speaker name spellings vary** ("Sohaib", "Suhail", "Sohil", "Suheel"; "Ankush Astogi" vs. "Ankush"), suggesting auto-transcription, not the speakers' own variability. ## Tone - **Conversational and mildly self-deprecating about the problem framing**: the trap *"looks harmless at first."* - **Instructional but non-alarmist**; problems are introduced as engineering realities rather than failures: *"It fails because every request is forced to carry the entire catalog."* - Occasional informal asides soften the technical content: *"This is an underrated benefit."* ## Vocabulary & Framing Devices - **Coining memorable labels** for the anti-pattern: - *"We are calling that as a fat agent."* - *"That is why we are calling it the 100 tool agent trap."* - **Reframing analogies** to familiar concepts: - *"Semantic routing is basically a rack for tools."* (i.e., RAG for tools, repeated later: *"This is basically a rag for tools."*) - *"We have used lazy loading, just-in-time compilation, and on-demand resource loading from years. We are just applying the simple and same principle to the LLM context."* - **Plain-language summaries after technical points**: *"So, in short, the idea is pretty simple."*; *"This is the core lesson from the benchmark."* ## Recurring Phrases & Fillers - **"Say for example" is the most frequent verbal tic**, appearing many times: *"like say for example, 10 tools"*, *"say, for example, uh 741 tools"*, *"say, for example, if you have 500 tools"*. - **"So, basically" as a recurring connector**: *"So, basically that approach works well"*, *"So, basically, this will lead to context overload."* - **"So" as an all-purpose sentence starter** throughout both speakers' turns, plus frequent "um/uh" fillers, consistent with an unscripted or semi-scripted talk. - Repeated framing of the pair **latency and cost**: *"So, two more reasons over here. First is the latency and the second is the cost."* ## Teaching Habits - **Numbers-first argumentation**, with specific figures repeatedly cited: - Accuracy: *"78% of the times"*, drops to *"around 40%"*, then *"a mere 13.6%"*, while the router *"stays above 83%"*. - Tokens: *"127,000 tokens"* vs. *"closer to about 1,000 tokens. That is roughly a 99% reduction."* - **Concrete worked examples** to illustrate routing behavior: - *"Find me a flight to New York next Wednesday"* → returns flight/calendar tools. - *"What is the weather in Paris right now?"* → *"Flights tools are not injected."* - **Explicit caveat-giving / anti-overengineering guidance**: - *"If you have fewer than 20 tools, a router may be unnecessary."* - *"do not over-engineer small systems."* - *"Routing generally pays off when the catalog is large enough..."* - **Actionable, numbered checklists**: the production checklist runs first through sixth, and the implementation pattern is presented as *"There are three steps in it."* - **External evidence cited to legitimize claims**: Anthropic's MCP write-up is mentioned twice (*"150k tokens down to 2,000, which is 98.7% token reduction"*), plus Berkeley Function Calling Leaderboard, Toolbench, and open-source projects. - **Negative-space teaching** — emphasizing what is *removed*, not just added: *"the router does not only add the right tool, it also removes the wrong tool from the model's choice set."* ## Structure & Signposting - **Roadmap given upfront**: *"we'll show why it breaks, what the numbers look like, and how semantic routing with just-in-time context can help us fix this problem."* - **Explicit slide transitions**: *"So, on the next slide, we'll be discussing about the implementation pattern."*; *"So, in the next slide, we will discuss implementation checklist."* - **Numbered takeaway summary at the end** ("First thing is... The second one..."), ending with a distilled thesis: *"the goal is not to make these agents more complicated... It's basically to stop forcing the model to reason over irrelevant tools."* ## Uncertainty Notes - Attribution of specific lines to Sohail vs. Ankush is based on transcript handoff markers ("Thank you, Suhail") and role descriptions; some segments are ambiguous. - Filler frequency and name misspellings reflect the transcript, which may not perfectly capture actual speech — patterns above are drawn only from what the transcript shows. # Position Record: "The 100-Tool Agent Is a Trap" — Sohail Shaikh & Ankush Rastogi, Prosodica ## Speakers - **Sohail (Sohaib) Shaikh** — Data Scientist at Prosodica; focus on applied AI, NLP, conversational intelligence, RAG systems; model and routing behavior. - **Ankush Rastogi** — Senior Data Solutions Engineer at Prosodica; focus on system design, implementation, and production trade-offs. ## Core Thesis - Giving an AI agent **all tool access at once** ("fat agent") works in demos and with ~10 tools, but degrades as the catalog grows: slower, more expensive, less accurate. - The failure is architectural, not tool quality: *"the design does not fail because one tool is badly written. It fails because every request is forced to carry the entire catalog."* ## Key Claims and Evidence ### The "Fat Agent" problem (Sohail) - At **741 tools**, tool descriptions alone consume **~127,000 tokens** before the user's question. - **Accuracy curve (their benchmark)**: fat agent ~78% at 10 tools → ~40% at 100 tools → **13.6%** at 741 tools (~1 correct in 8). - Semantic router stays **above 83%** across the same catalog sizes, because the model chooses from a small relevant set. - Attribution of cause: the **"lost in the middle"** problem — models attend more to beginning/end of long context; hundreds of schemas packed in the middle are not reliably used. - **Latency**: at 500 tools, fat agent can push time-to-first-token past **5 seconds**; router stays near-flat. - **Cost**: just-in-time routing reduces tool context to ~1,000 tokens (3–5 schemas) — ~**99% reduction**. ### Caveat (Ankush) - **< 20 tools**: router may be unnecessary; load tools directly. - **> 50 tools in production**: router-based schema selection is justified. ### Semantic routing mechanism (Sohail) - Described as **"RAG for tools"**: embed tool descriptions offline into a vector index; at runtime embed the query, retrieve top-K tools (typically 3–5), inject only those schemas. - Runtime overhead: one embedding call + one vector search. ### Just-in-time context injection (Ankush) - Semantic routing = retrieval layer; JIT context = context management strategy. - Frames it as analogous to lazy loading / JIT compilation. - Cites **Anthropic's on-demand tool loading via MCP**: 150k → 2k tokens (**98.7% reduction**). - Evaluation methodology: Berkeley Function Calling Leaderboard, Toolbench-style scenarios, synthetic tool pools at 10/50/100/200/1041 tools; same model, answer key, and catalog in both modes; K swept at 3, 5, 10. **K=5 is their recommended default.** - Example walkthrough: "Find me a flight to New York" → router injects only flight/calendar tools; router also **removes wrong tools** from the choice set. ### Community corroboration (Ankush, as reported) - Anthropic MCP write-up; developer issues in mainstream agent/SDK projects on latency/token bloat/tool confusion; open-source **MCP-zero** exploring routing at thousands of tools; forum posts of tool confusion below 100 tools. - Position: if an agent degrades as tools are added, it *"may mean the architecture is asking the model to solve the wrong problem."* ### Named ecosystem resources (Ankush) - Aurelio Labs' open-source semantic router, Toolbench, Berkeley Function Calling Leaderboard, Anthropic MCP write-up, vector stores (Pinecone, Qdrant, Chroma). ## Trade-offs and Risks (stated by Ankush) - **Router miss**: mitigate via fallback — widen K, second retrieval pass, or route to broader tool group. - **Weak descriptions**: embeddings inherit weakness; write descriptions in user language, including intent, action, key entities. - **Rare tools**: may never score high; monitor misses and rewrite descriptions. - **Over-engineering warning**: with 10–15 tools, static loading is fine. ## Takeaways (as expressed) 1. Tool overload hurts accuracy and adds cost/latency. 2. Semantic routing = RAG for tools; index descriptions, retrieve at runtime, inject only what's needed. 3. Start simple: K=5, log every decision, evaluate against a real test set, iterate on descriptions. 4. Goal is not complexity — it is *"to stop forcing the model to reason over irrelevant tools."* ## Qualifications Preserved - Benchmark numbers are the presenters' own; Anthropic's 150k→2k figure is reported as a third-party citation. - The 78%/40%/13.6% accuracy figures shift context (741 vs. 1041 tools) between slides; recorded as stated without smoothing. - Router benefit is explicitly conditional on catalog size — not presented as universally better. # Speaker Attribution Record **Video:** The 100-Tool Agent Is a Trap - Sohail Shaikh & Ankush Rastogi, Prosodica **Video ID:** 01a15ca8-e40d-7df0-ab87-fc53af99f3af ## Speakers | Speaker | Role | Self-Identified Details | |---|---|---| | **Sohail (Sohaib/Suhail/Suheeh) Shaikh** | Co-presenter / creator | Data scientist at Prosodica; background in AI, NLP, marketing, analytics, engineering; focus on applied AI, NLP, conversational intelligence, RAG systems | | **Ankush (Ankur) Rastogi / Astogi** | Co-presenter / creator | Senior data solutions engineer at Prosodica; 10+ years in AI, data engineering, production systems; covers system design and production trade-offs | - **Role division (stated by speakers):** Sohail covers model and routing behavior; Ankush covers system design, implementation, and production trade-offs. - **No narrator, guest, or other third-party speakers** appear in the transcript. > Note: The transcript contains multiple phonetic spellings of both names (Sohaib/Suhail/Suheeh; Astogi/Astogi/Ankur); the title metadata gives "Sohail Shaikh & Ankush Rastogi." ## Referenced Sources & Third-Party Claims - **Anthropic MCP write-up:** Cited twice — claims on-demand tool loading reduced token usage from 150k to 2,000 tokens (98.7% reduction). Presented as Anthropic's reported figure, not the presenters' own. - **Berkeley Function Calling Leaderboard:** Used as evaluation dataset; also recommended as a starting point. - **ToolBench / "Skills bench style scenarios":** Used as evaluation datasets. - **Synthetic tool pools:** Created by presenters to scale tool counts (10 to 1041). - **Areilo Labs semantic router:** Open-source project recommended for local testing/benchmarking. - **MCP Zero:** Open-source project noted as exploring routing at large scale (thousands of tools). - **Community/forums:** Presenters claim developers have raised similar issues in mainstream agent/SDK projects and forum posts; no specific sources named. - **Vector stores mentioned:** Chroma DB, Pinecone, Qdrant ("Quadrant"). ## Presenters' Own Claims (Benchmark Results) - Fat agent accuracy: ~78% at 10 tools → ~40% at 100 → 13.6% at 741/1041 tools. - Semantic router accuracy: stays above 83% across catalog sizes. - 741 tools ≈ 127,000 tokens of tool context per request; routed prompts ~1,000 tokens (~99% reduction claimed). - Fat agent latency: >5 seconds time-to-first-token at 500 tools (presenters' claim). - Recommended default K = 5; router unnecessary below ~20 tools, justified past ~50. ## Promotional Segments - **Closing LinkedIn QR codes:** "You can scan our LinkedIn QR codes on this slide to connect with us" — flagged as promotion/connect CTA. - **Share request:** "please share it with someone building agents" — promotional CTA. - No product or paid-service promotion identified beyond employer affiliation (Prosodica) and open-source references. ## Metadata & Availability Limits - No curated metadata supplied beyond title and transcript. - **Comment availability:** Not stated in source; unknown. - No timestamps present in transcript; segment ordering only. Hi everyone. Thanks for being here. I am Sohaib here and along with me is Ankush. So, today we'll be talking about a mistake that looks harmless at first. Giving which is basically giving an AI agent every tool access it might ever need all at once. So, basically that approach works well in a demo. It might even work with a small number of tools like like say for example 10 tools. But once the catalog grows, the agent gets slower. It might become more expensive and less accurate as well. That is why we are calling it the 100 tool agent trap. In the next half an hour or so, we'll show why it breaks, what the numbers look like, and how semantic routing with just-in-time context can help us fix this problem. So, a quick introduction about myself. I am Sohaib Shaikh. I'm currently working as a data scientist with Presodica. My background spans across AI, NLP, marketing, analytics, and even engineering. My current focus is on applied AI, NLP, and conversational intelligence along with rack systems. I'm especially interested in making AI systems more reliable, measurable, and even scalable beyond the demos in production. \>\> And I'm Ankush Astogi. I work as a senior data solutions engineer at Presodica. I have spent more than a decade in AI, data engineering, and production systems. My focus is the engineering side. So, it's not about what's going to work in notebook, but whether it's going to survive in with real load, real user, and real failures. So, that is the angle we are taking today. Suhail will focus more on the model and routing behavior, and I will focus more on system design, implementation, and production trade-offs. \>\> Awesome. Thank you, Ankush. Um so, let's get into it. So, let's imagine a common design. Um you tend to build a system, and it can do many things. Say, for example, querying a database, sending an email, um even checking a calendar, or looking up an order, calling an API, and so on and so forth. The simplest approach over here would be to give a model every tool definition on every request. Every function name, um every description, and even every JSON schema will go into the prompt, whether the user might need it or not. So, that We are calling that as a fat agent. At small scale, it feels fine. With 10 tools as well, the model might usually pick the right one. The demo looks good. Then, the product grows. Um 10 tools might become 30, or it will keep on increasing, and eventually the model starts calling the wrong function, and starts confusing similar tools, may invent tool names, and even take longer to respond. The important point is, basically, the design does not fail because one tool is badly written. It fails because every request is forced to carry the entire catalog. So, let's look here. There are, say for example, uh 741 tools in your uh in your entire schema, but and it will look, basically, take up to 127,000 tokens just to have all those tool descriptions in it. And this is even before the user user's actual question is even considered. So, basically, this will lead to context overload and uh we need to manage that properly. So, on this slide we see why it is failing and why the accuracy collapses beyond a point. So, when you look at the accuracy curve with the 10 tools, fat agent will get the tools right almost 78% of the times. That is not perfect, but it's usable. At almost 100 tools, the accuracy drops to around 40%. Less than half of the tools that are called are the correct tools. And if it grows beyond that, like say for example, in over here, at 741 tools, the accuracy will be a mere 13.6%. So, in short, it's roughly one correct tool out of eight tools. So, when we compare it with the semantic router, semantic router behaves very differently. It stays above 83% across the same catalog sizes. That is because the model is not choosing from hundreds of tools, it's choosing from a small and relevant set. One reason the fat agent fails is because it's lost in the middle problem. Model pays stronger attention to the beginning and end of the long context. When hundreds of tool schemas are packed into the middle, the model does not reliably use them. So, we end up paying paying for a huge prompt, and that prompts makes the decisions even harder. So, two more reasons over here. First is the latency and the second is the cost. So, we saw in the earlier slide that accuracy was a big problem. Another issue is with latency and cost over here as we can see. So, like say for example, if we have 741 tools, we saw that it requires almost 127,000 tokens. That will include the tool description and the schema text. So, that cost is basically being paid on every request. At 100,000 requests a day, if you push this into production, you are sending in billions of tokens and just to describe those tools. And with just-in-time routing, the prompt may include only three to five relevant schemas, closer to about 1,000 tokens. That is roughly a 99% reduction in tool context tokens. Another issue is latency. With fat agent approach, time to first token grows as the tool catalog grows because the model has to process a larger prompt before it can answer the user question. So, say for example, if you have 500 tools in your agent, the fat agent path can push first token latency past 5 seconds. So, this is specifically important if your product is real-time because this will take longer time to respond. So, which will make the design feel slow and unpredictable. So, and with semantic routing, it might feel more responsive and it will be more real-time in a way. \>\> Here is the clean comparison. So, on the left side, we have a fat agent design and on the right side, we have our semantic router design. So, in the fat agent design, every schema is loaded for every request, no matter what the user asked. And because of that, the catalog will grow, the prompt will grow, due to which the latency will rise and accuracy will drop. The agent also becomes one large monolithic, which makes it hard to test, risky to update, and even painful to debug. But, in the case of semantic routing design, the agent does not start with every tool. So, in this case, the router first look at the user's query. It retrieves the three to five most relevant tool, and then it injects only those into the model call. So, the context stays small, latency stays stable, and the accuracy holds because the model is choosing from a focused list instead of giant catalog. This is Okay, so there is a one important caveat here. If you have fewer than 20 tools, a router may be unnecessary. Just load the tools directly. But, once you passed 50 tools in the production system, then justifying router-based schema make more sense. \>\> Thanks, Ankush. So, let's now look into how semantic routing works. If you have built a rack system previously, this will feel pretty similar to that. The difference is that we will retrieve tools instead of the documents. So, at the first stage, say for example, over here, each tool will have a clear description. For example, a search flights tool, uh a check calendar availability tool, or a retrieve customer order status. Say for example, the second step would be those descriptions needs to get embedded and stored in a vector index. This is usually done offline when the tool catalog is created or updated. At the runtime, the user will ask a question, we embed the user query with the same embedding model. Then, we search that vector index for the tools with the description that are closest to the query. So, the router will then return the top K tools. Often, K would be three or five tools that matches the user query. And only those selected schemas will get injected into the model call. So, this is the whole pattern. At the first step, you need to index tool descriptions offline. Then, retrieve relevant tools at runtime and keep the model's context focused. So, in short, the idea is pretty simple. Semantic routing is basically a rack for tools. Um if you already have an embedding model and vector database in your stack, most of the infrastructure is already familiar to you. \>\> Thank you, Sohil. Now, we will deep dive into just-in-time context injection. So, semantic routing is the retrieval layer. And just-in-time context injection is the context management strategy. So, in the case of fat agent approach, it loads everything before the query is understood. Due to which the model receives a huge list of tools first, and then it tries to reason through it. Just-in-time context does the reverse. It will wait until the query is known, then it will inject only the context needed for that request. I mean, this is not a new software idea. We have used lazy loading, just-in-time compilation, and on-demand resource loading from years. We are just applying the simple and same principle to the LLM context. Anthropic has written about this pattern with on-demand tooling loading through MCP. Their report token usage went from 150k tokens down to 2,000, which is 98.7% token reduction. That is the signal. Large tool catalog should not be dumped into the into every prompt. They should be retrieved when needed. So, moving to the next slide, let's talk about how to evaluate this fairly. We measure four things here: tool selection accuracy, time to first token, input tokens per request, and the estimated cost per thousand call. So, here for data sets, we used Berkeley function calling leaderboard, Skills bench style scenarios, and synthetic tool pools that allow us to scale the number of tools, so we can test at 10, 50, 100, 200, or even 1041 tools. So, we ran the same queries in two modes, the fat agent one and the semantic routing. Same model, same answer key, same tool catalog. The only difference was whether the model saw every tool or only the routed tool. We also swept K at 3, 5, 10 to understand the trade-off. So, the smaller K is the faster and cheaper. Larger K can be the can recover more edge cases. In practice, K equals 5 is a strong default starting point. So, if you are want to try this, you can default it to five. Here, K means the number of tools the semantic router retrieves and gives to the LLM for each user queries. Here are the benchmark result that we got. So, here is the result on the left side. We see the fat agent accuracy line drop sharply as the tool count grows. It started around 78% at 10 tool and then falls to around 13% at 1041 tools. But, in the case of router, the router line stays above 83% because the model stays stable. From the model points of view, it always choosing the handful of tool, even if the real catalog contains 100. On the right side, time to first token shows the same story. The fat agent path gets slower as more tool schemas are added. At large catalog sizes, the model spend noticeable time just processing the prompt. But in case of router, it stays almost flat because of the prompt size is under controlled. This is the core lesson from the benchmark. The catalog can grow, but the model's working set should stay small. Now, on the next slide, we'll be discussing about the implementation pattern. There are three steps in it. Step one is offline. So, build a catalog of tool. For each tool, store the name, description, schema. Then, embed the description and store it into a vector database. You can use Chroma DB, you can use Pinecone, you can use Quadrant. Whichever vector database you are using. Then, step two will happen on every request. Embed the user's query. Then, run the nearest neighbor search. And return the top K tools. In the step three, it will also happen on every request. Fetch the schema for those selected tools. And then, pass only those schemas into the model call and log which tool was selected. So, here, logging really matters. So, let's say if you miss something and if you want to improve the description or you want to tune the K, you need a good logging system that you can use it. So, the runtime overhead is small, one embedding call, and one vector search. The payoff is much smaller prompt and more stable tool selection problem. \>\> Awesome. So, now that you have learned about the implementation pattern, I'll take you through the code version of it, which is pretty straightforward. So, initially you need to loop through the tool catalog, and for each tool, as Ankur mentioned, embed the description, store that in the vector database with the tool name and other details. At run time, embed that user query, compare that query vector against the stored tool vectors using cosine similarity or vector search, and then retrieve the top most relevant tools, and then call the model only with those schemas and the function or tool parameter. That final line is the most important part because the model does not receive all the tools, it only receive the routed tools. This works with any embedding model and at and with any vector database. You can start, as Ankur mentioned, you can start with Quadrant or any other database vector database locally, and then move to a more managed vector store later if needed. If you already have an existing rag infrastructure, this is not a new infrastructure as we mentioned earlier, it is the same retrieval pattern applied to the tool selection. \>\> Thank you, Suhail. Uh Now, we will discuss semantic routing in action. So, imagine you have a 200 tool agent. A user asks, "Find me a flight to New York next Wednesday. The router embeds that query. Returns tool like search flights, book flight, calendar check. The model sees only those relevant schema, so it is much more likely to call the correct tool. But, if we load all 200 tools, hotels tool, weather tool, emails tool, sequels tool, or any unrelated workflow tool, it would all be competing for attention. That is when model starts choosing the wrong function. Now, let's take a second query. What is the weather in Paris right now? The router returns get weather and get forecast. Flights tools are not injected. Hotels tools are not injected. They are simply not available at this request. This is an underrated benefit. But, the router does not only add the right tool, it also removes the wrong tool from the model's choice set. Now, in the next slide, we will discuss implementation checklist. So, here is the production checklist if somebody wants to uh follow this along. So, first, get locked the tool in one place. So, add name, description, schema, owner, and the version. Second is to build the index. Embed each description and store the vectors. The third one is write the router. So, in this, embed the query, then search the index, and take top K and fetch schema. The next one is the fourth one in which a user need to wire it up into the agent loop. So, the model tool list should come from the router, not from a hard-coded full catalog. Fifth is to evaluate. Run your test set at K equals to three, five, and 10, and then pick the smallest K that meets your accuracy target. Sixth is to monitor production. Log selected tools, final tool calls, failure, and fallback usage. Then re-embed tools when descriptions or schema change. This is not the 6-month platform rewrite. For most team, it is a focused sprint. So, in the next slide, we will discuss what community already has confirmed. This is not just the observation. So, Anthropic has published on-demand tool loading with MCP, and reported a drop of 150k tokens to 2k, which is a huge reduction in tokens. Developers have raised similar issues in mainstream agent and SDK projects, pointing out that sending all tools definition on every request can increase latency, token usage, and tool confusion. Open proj open-source projects like MCP zero have also explored routing at very large scale, including thousands of tool across many servers. And you can find form forums post from builder who hit tool confusion with only few tools. Long before they reach 100. So, if you if your agent start failing as tool are added, it doesn't automatically means your prompt are bad. It may means the architecture is asking the model to solve the wrong problem. \>\> So, here we are just trying to say about the We are going to talk about the open-source projects that we came across. So, one of them is of from Areilo Labs. They have an open-source semantic router project that you can easily use to test locally for benchmarking, Toolbench, and the Berkeley function calling loader leaderboard are also useful starting points. And for practical guidance, you could read the Anthropic MCP write-up. It explains why on-demand tool loading matters and gives a strong real-world signal for this architecture. The point is that you don't need to invent the ecosystem from scratch. Those pieces already exist. Next slide. Sorry. Here it is. Thank you. So, there are a few trade-offs with this. The first risk could be a router miss. So, if the So, the router may fail to retrieve the tool the models need and handle that with you could handle that with a feed fallback. If the model cannot complete the task, you could widen K, run a second retrieval pass, or even route it to a broader tool group. The second risk would be weak tool descriptions. If your descriptions are weak, embeddings will end up being weak. Write descriptions in the words users actually use and include intent, action, and key entities along with it. The third risk would over here would be rare tools. So, some tools may never score high unless their descriptions include the right language. So, you need to monitor data misses and rewrite those descriptions accordingly. And finally, just to conclude, do not over-engineer small systems. If you have 10 or 15 tools, static loading may still be fine. Routing generally pays off when the catalog is large enough that the prompt size, latency, or tool confusion becomes a real production problem. So, a few takeaways. First thing is, as we discussed, tool overload hurts accuracy. Loading every schema into every prompt makes the model's decision harder as the catalog grows on. The second one would be to about tokens. It's like tokens are both cost and latency. Larger tool catalogs can add add tens of thousands of tokens before the user's actual request is handled. Semantic routing can help fix that problem. The When the catalog can be is large, the model only sees the tools relevant to the current query. This is basically a rag for tools. Index those descriptions, properly retrieve at run time, and inject only what is needed. And finally, start simple. Use K as five as a starting point, log every decision, and evaluate against a real test set. And improve their tools descriptions over time in case you get incorrect tools being picked up. And the goal is not to make these agents more complicated. It's It is basically to stop forcing the model to reason over irrelevant tools. So, there are some references on this slide. So, if you're building something, you can use these references. Like semantic routing libraries, tool bench, Berkeley function calling leaderboard, Anthropic MCP write-up, and vector stores like Pinecone, Quadrant. Thank you so much for watching this. If this was useful, please share it with someone building agents. They have either already hit this wall or they're about to. You can scan our LinkedIn QR codes on this slide to connect with us. We are happy to continue the conversation, share resources, or hear how others are approaching this problem in their own system. Thank you so much for watching this. I'm Ankush. This is Suheel, here. Thank you so much. Yeah. Thanks, everyone.