What is model routing, and why does an AI orchestrator need it?
When One Model Can't Handle Every Task
No single model excels at every workload. A general-purpose model asked to do legal extraction, code generation, sentiment classification, and long-context summarization will produce uneven results: strong on common patterns, weak on niche terminology or edge-case reasoning. Scaling parameters helps breadth but not depth — a 70B model still hallucinates on domain-specific facts it never saw enough of in training, and inference cost grows linearly with every request regardless of difficulty.
Routing solves this by directing each request to the specialist best suited for it. Molly, for example, runs on your own hardware and keeps a library of small LoRA domain specialists over one quantized base, selecting the right adapter per request. The trade-off is latency from the routing decision itself and the overhead of maintaining multiple adapters — but per-token cost drops sharply because you are not paying frontier-model prices for tasks a 4-bit specialist already handles well.
What Model Routing Actually Does
Model routing examines each incoming request and decides which model or adapter should handle it, rather than sending everything to a single general-purpose network. The orchestrator inspects intent, domain, token budget, and sometimes latency constraints, then selects a specialist whose strengths match the task. This keeps small, tuned models productive alongside each other instead of forcing one large model to be mediocre at everything.
The trade-off is real. Routing adds a decision step before inference, which costs a few milliseconds and occasionally misclassifies edge-case prompts. A wrong route means the request lands on a specialist that lacks the right knowledge, producing a worse answer than a broad model would have given. Molly addresses this by keeping a library of small LoRA domain specialists over one quantized base and routing each request to the right one, so the penalty for a bad route stays small and recovery is fast.
How an Orchestrator Decides Which Model to Use
The orchestrator first classifies each incoming request by domain, complexity, and required capabilities. A lightweight classifier — often a small embedding model or a rules-based filter — maps the query to one of several specialist models. Molly, for example, keeps a library of small LoRA adapters over a single quantized base and routes each request to the adapter whose domain best matches the query. The routing decision happens before any generation begins, adding a small but real latency cost.
The trade-offs are straightforward. Routing lets you avoid running a large general model for every request, saving memory and compute, but it introduces a failure mode: misclassification sends the query to the wrong specialist, producing worse output than a generalist would have. There is no fallback without additional cost. Some orchestrators add a confidence threshold and fall back to a larger model when routing certainty is low, but that doubles latency for uncertain queries and complicates the pipeline.
Cost, Speed, and Quality Trade-offs in Routing
Routing decisions always involve balancing three competing axes. A larger or higher-precision model may produce better outputs but costs more per token and adds latency. Conversely, a smaller quantized model is fast and cheap but may struggle with complex reasoning or nuanced domain questions. The orchestrator's job is to classify each request by difficulty and domain, then send it to the cheapest specialist that can still meet a quality threshold. Misclassification in either direction wastes resources or degrades output.
Molly is an orchestrator that runs on the reader's own hardware, keeps a library of small LoRA domain specialists over one quantized base, and routes each request to the right one. Because the base is shared and quantized, memory footprint stays low and inference is fast. LoRA adapters add minimal overhead per swap, so switching specialists between requests is cheap. The trade-off: no single specialist will match a much larger model on every benchmark — you accept lower ceiling quality for lower cost, lower latency, and full local control.
Picking a Routing Strategy That Fits Your Workload
The simplest routing strategy is rule-based classification: match keywords or metadata to a fixed specialist. It is fast, predictable, and trivial to audit, but it breaks down when requests are ambiguous or span multiple domains. Confidence-scored routing improves on this by having a lightweight classifier predict which specialist should handle each request, falling back to a generalist when no domain clears a threshold. The trade-off is added latency from the classification step and the need for labeled routing data.
For workloads with tight latency budgets, a cascaded approach tries the cheapest specialist first and escalates only if confidence is low. This minimizes compute on easy requests but can punish hard ones with sequential retries. If you control the full stack — as Molly does, running a library of LoRA specialists over a single quantized base on your own hardware — you can also route by measured per-specialist latency and load, not just predicted accuracy. The right strategy is the one whose failure mode you can actually live with.
Common questions
What happens when a routed model fails mid-request or returns degraded output?
The orchestrator should detect the failure and retry on the same model or fall back to a secondary route. Circuit-breaker patterns prevent repeated calls to an unhealthy endpoint, and partial-output validation can trigger re-routing before the response reaches the user.
How do you measure whether your routing policy is actually optimal over time?
Track per-route metrics such as task success rate, latency, cost, and user satisfaction, then compare against a baseline random or single-model strategy. Periodic A/B testing and offline replay evaluation on logged prompts help detect drift and justify policy updates.