The default that stopped being defensible
For most of the LLM era, the default architecture for a new AI feature was the same: pick a frontier model, send everything to it, iterate on the prompt until it works. Simple, fast to ship, and until recently, perfectly defensible. The frontier model was smart enough to handle almost anything, and the cost was low enough not to matter at early-stage volumes.
That calculus has quietly broken. With the price gap between frontier models and well-tuned small models widening, and the capability gap on routine tasks narrowing, running everything on your biggest model is now leaving money on the table. Sometimes a lot of money.
The response, across virtually every team we've worked with in the past year, has been the same: build a router.
What model routing actually means
Model routing is the practice of sending different requests to different models based on what each request needs. At its simplest, you have two models — a cheap one and an expensive one — and a decision function that picks which to use. In more mature setups, you have four or five models and a routing layer that considers the task type, the input complexity, the latency budget, and sometimes even the user's plan tier.
The payoff is immediate: for most real-world traffic, 70–90% of requests are "easy" and can be handled by a smaller, cheaper model with no quality degradation. Only the remaining 10–30% need the frontier model. Routing correctly can cut your inference bill by 60% or more without users noticing anything.
The three styles of router
Rule-based routing
The simplest router is a set of hand-written rules. Summaries under 500 words go to the small model. Complex reasoning tasks go to the big model. Code generation goes to the code-specialized model. Rules are transparent, easy to debug, and require no ML infrastructure — which is exactly why most teams should start here.
The downside is that rules don't generalize. Every new task type requires new rules, and rules that worked six months ago may not reflect current model capabilities as providers ship updates.
Classifier-based routing
A step up: train a lightweight classifier to predict which model a request should go to. The classifier takes the input and outputs a routing decision. These classifiers are cheap to run — often sub-10ms per request — and they learn patterns that would be tedious to encode as rules.
The challenge is training data. You need labeled examples of "this request should go to model X" to train the classifier, which means you either bootstrap from rules or run all requests through multiple models and use the results to generate labels. Both approaches work; neither is cheap to set up.
LLM-as-router
The most flexible option: use a small LLM to make the routing decision itself. You prompt it with the request and a description of available models, and it picks the best fit. This handles edge cases gracefully but adds noticeable latency and cost to every request.
In practice, LLM-as-router makes sense for high-value tasks where getting the routing wrong is expensive, and where the latency hit is acceptable. It's usually overkill for high-volume consumer applications.
The cascade pattern
A more powerful variant of routing is the cascade: try the cheapest model first, and if the result isn't good enough, escalate to a more expensive one. This works when you can cheaply evaluate whether the cheap model's answer is acceptable.
Typical escalation signals include:
- Low confidence scores from the model
- Schema or format violations in the output
- Failed consistency checks against a ground truth
- User feedback or explicit retries
Cascades capture most of the value of aggressive routing without requiring you to perfectly predict which requests are hard. You let the system discover that empirically. The tradeoff is that a cascade's worst-case latency is higher — escalated requests pay the cost of both models — so it fits better in asynchronous workflows than interactive ones.
What to measure
A routing system without good telemetry will silently degrade over time. At minimum, track:
- Cost per request, broken down by route — the thing you're optimizing
- Quality per route — against a task-specific eval set, run continuously
- Latency distribution per route — because users care about p95, not average
- Escalation rate — for cascades, how often does the cheap model fail?
- Routing confidence — for classifier-based routers, are the decisions clear or marginal?
The metric that matters most is the quality-to-cost ratio per task. A router that saves 80% of your cost but drops quality by 20% is usually a bad trade. A router that saves 40% with no measurable quality drop is a great one. Know which one you're shipping.
Where this is going
Routing is becoming a standard layer in production LLM stacks, in the same way that load balancers became a standard layer in web infrastructure. The tools for doing it well are maturing rapidly — most commercial LLM gateways now include routing primitives out of the box, and open-source frameworks are catching up.
The question for most teams is no longer whether to route, but how much of the routing logic to own themselves versus delegate to a gateway. Either answer can be right. The wrong answer is not routing at all, because your competitors are, and their unit economics are getting better while yours stand still.