The ceiling everyone hits
There's a threshold in agent development that almost every team discovers the same way. You start with five or six tools. Everything works beautifully. You add a few more. Still fine. Somewhere around fifteen tools, things start getting weird — the model picks the wrong tool, forgets tools exist, or calls them with parameters that mix up two different interfaces. By thirty tools, the agent is noticeably worse than it was with ten.
This isn't a model quality problem. It's a context saturation problem. Every tool definition takes hundreds of tokens, and the model's ability to reliably select from a menu degrades as the menu gets longer. The fix isn't a better model — it's not loading all the tools in the first place.
The core idea
Dynamic tool retrieval works like this: instead of putting every available tool in the system prompt, you put the tools in an index. When the agent starts a task, a retrieval step selects the most relevant subset — typically five to fifteen tools — based on the user's request. Only that subset goes into the model's context.
The agent never sees the full catalog. It sees exactly the tools likely to be useful for the current task, and nothing else.
This pattern has been quietly emerging across major agent frameworks, and OpenAI's recent Tool Search feature in GPT-5.4 made it official: dynamic tool loading is now a first-class capability, not a workaround.
How the retrieval actually works
Embedding-based tool selection
The most common implementation embeds each tool's name, description, and example usage into a vector store. When a new request comes in, you embed the request and retrieve the top-k closest tools.
This works surprisingly well when tool descriptions are written carefully — think of them as short docstrings optimized for semantic search, not just API reference material. Tools with vague names and terse descriptions retrieve poorly.
Classifier-based routing
For systems where latency matters more than flexibility, a lightweight classifier can map request types to tool categories. "Billing question" routes to the billing tools, "data request" routes to the query tools, and so on. This is faster than embedding lookup but requires maintaining the routing logic.
Hierarchical loading
The pattern that scales best in our experience is hierarchical: the agent first selects a category of tools, then the specific tools within that category. This matches how humans navigate large APIs — you don't memorize every endpoint, you learn where things live.
What this unlocks
Scaling to hundreds of tools
Systems we've seen in production are now running with 200+ tools behind a dynamic retrieval layer, and the agent performance is actually better than it was with 30 tools loaded statically. Less context means more attention on the tools that matter.
Multi-tenant tool access
Different users can have different tools available based on permissions, subscription tier, or team — without needing separate agent deployments. The retrieval layer becomes the access control point.
Graceful capability expansion
Adding a new tool no longer requires re-tuning the whole system prompt or re-running evals on the full tool set. You add the tool to the index, write a good description, and the agent starts using it when it's relevant.
The tradeoffs you need to understand
Dynamic retrieval is not free. It introduces an extra step before every agent turn, which adds latency — usually 50–200ms depending on the retrieval method. For high-frequency interactions, that cost adds up.
More importantly, retrieval can fail. If the right tool isn't in the top-k results, the agent will try to solve the task with whatever tools it did get, which often leads to creative failures. Mitigations:
- Track retrieval recall. Log every request along with the tool the agent actually used, and measure how often the "right" tool was in the retrieved set.
- Provide a fallback search tool. Let the agent explicitly search for tools it doesn't see but suspects exist. This gives you a safety net when retrieval misses.
- Tune the k value empirically. Five tools is often too few; twenty is often too many. The sweet spot depends on your domain.
The MCP connection
Dynamic tool retrieval is a natural fit for MCP-based architectures. Because MCP provides a standard tool-discovery interface, you can build a retrieval layer that indexes tools from multiple MCP servers without knowing anything about their internal implementations. The result is an agent that can compose capabilities across different services on demand — without anyone having to manually decide which tools to expose.
This is part of why MCP adoption has accelerated so quickly: it's not just a protocol, it's the substrate for the next generation of agent tooling.
Start simple
If you're hitting the tool-count ceiling, don't jump straight to a sophisticated retrieval system. Start with the simplest version that could work: embed tool descriptions, retrieve top-5 on each turn, measure what breaks. Ninety percent of the benefit comes from the first implementation. The last ten percent is where the interesting engineering lives — but you'll only understand what that interesting engineering needs to look like after you've shipped the boring version.