Look at a week of real production traffic through an LLM feature and sort it by difficulty. The distribution is almost always the same shape: a long, fat head of requests that are trivial, repetitive, or already answered, and a thin tail that genuinely needs everything your most capable model can do.

Now look at which model handled each of them. For most teams, the answer is “the same one,” and it is the expensive one. That single decision — made early, usually for good reasons, and then never revisited — is the most common unforced error I find in production LLM systems.

What a router actually is

A routing layer is not complicated. It sits between your application and your model providers and does three things: it classifies the incoming request, it applies a policy to pick a model, and it defines the conditions under which the result gets escalated to something stronger.

The classification does not need to be clever. In most systems, request type is already known by the calling code — a summarisation call and a code-generation call arrive at different endpoints. Half the benefit of routing is available from static rules written by someone who understands the product.

Build the evaluation set first

This is the step that makes routing safe, and it is the step teams want to skip. Before anything is rerouted you need to know what “good enough” means for each request type, measured against real examples from your own traffic.

Two or three hundred representative requests with expected outputs is usually enough to start. Label them by request type. Establish what your current model scores. That baseline is what converts routing from a gamble into an experiment: you can now answer “did quality drop?” with a number rather than a feeling.

Without it, cost optimization is just quality degradation with better marketing — and you will find out which one you did from a customer.

Three strategies, in order of effort

Static rules. Route by request type, input length, or customer tier. Crude, transparent, and frequently captures the majority of the available saving. Start here. If classification-type requests go to a small model and open-ended generation goes to a large one, you have already done most of the work.

Cascade with escalation. Try the cheap model first. Check the result against a validator — schema conformance, a confidence signal, a rules check, or a small grader model — and escalate to the stronger model only on failure. You pay for two calls on the escalated fraction, so the economics depend on the escalation rate. Below roughly 30 per cent this is usually a clear win; above it, the double-calling starts eating the benefit.

Learned classification. Train a small classifier on your labelled traffic to predict which model is needed. More accurate, and more moving parts: the classifier needs monitoring and periodic retraining as your traffic shifts. Worth it at high volume, premature below it.

Cache before you route

Routing gets the attention, but caching is often the bigger and cheaper win, and it composes with everything else. Exact-match caching on identical prompts costs almost nothing to implement and is pure saving. Semantic caching — matching on embedding similarity above a threshold — extends the hit rate to near-duplicate requests, at the cost of needing a carefully chosen threshold and a way to detect bad hits.

Where your provider offers prompt caching for long, stable system prompts, turn it on before doing anything else. It is a configuration change with a direct discount attached.

What to measure afterwards

Three numbers tell you whether the router is working:

  • Cost per request, tracked per request type rather than in aggregate, so a regression in one category cannot hide inside an improvement in another.
  • Quality regression rate against your evaluation set, run on every routing policy change. This is the number that protects you.
  • p95 latency, because smaller models are usually faster and escalation adds a round trip. Routing often improves the median and worsens the tail.

Watch the escalation rate too. If it climbs over time, your traffic is drifting away from the assumptions the policy was built on, and the thresholds need revisiting.

When routing is the wrong answer

Routing adds a component, and components fail. It is not free, and there are cases where it is not worth it:

  • Low volume. If your monthly model spend is in the low hundreds of dollars, the engineering time costs more than the saving. Fix the context bloat instead and move on.
  • Uniformly hard traffic. Some workloads genuinely need the frontier model for every request. Measure before assuming yours does — but if it does, accept it.
  • Regulated or high-stakes decisions. Where being wrong is expensive in a way that is not measured in dollars, the cheapest model that clears the bar is the wrong optimization target.
The right model for a request is the smallest one that clears your quality bar. The only way to know which one that is, is to have written the bar down.

Most teams reach for a bigger model when quality disappoints them. Fewer ask whether the expensive model is being wasted on the eighty per cent of traffic that never needed it. That second question is usually where the money is.