Model Routing Patterns: Sending Requests to the Right Model
A single model for everything is expensive and slow. A routing layer that picks the model per request is the architecture that scales.
Using one model for every request is like using a freight truck for every delivery — it works, but the grocery run is expensive. Model routing is the pattern of classifying each request and sending it to the model that fits: cheap and fast for simple tasks, frontier and capable for hard ones.
The router is not the model
The router is a classifier. It can be a small model, a rules engine, or an embedding-based similarity match. Its job is to label the request with a complexity tier and route accordingly. The router itself must be cheap — if the router costs as much as the request, you have added latency without saving money.
Routing by task type
Classification, extraction, formatting, and summarization are tier-1 tasks — a small model handles them at a tenth of the cost. Multi-step reasoning, tool use, and creative generation are tier-3 — frontier models. Tier-2 is the middle: drafts that a small model starts and a frontier model refines.
The fallback problem
When the tier-1 model fails — the output is wrong or the request was misclassified — you need a fallback to the tier-3 model. But detecting failure is itself a classification problem. A confidence threshold on the small model, or a quick validator on the output, is the signal. Without a fallback, routing saves money at the cost of quality; without a validator, it degrades quality silently.
Cost visibility
Tag every request with the model that handled it and the cost. Without per-route cost breakdowns, you cannot tell whether the router is saving money or just adding complexity. The dashboard should show: tier-1 hit rate, fallback rate, cost per tier, and the misclassification cost — requests that went to tier-1, failed, and retried at tier-3.
