Right-sizing models: routing requests by complexity

The question

Most requests to an AI service are simple. A large model can answer them, but so can a much smaller one, at a fraction of the cost and energy. The question for a sovereign service is how much can be saved by sending each request to the smallest model that answers it well, and what that costs in quality.

A simple model of routing

Suppose a router sends a share $p$ of requests to a small model and the rest to a large one. With $c_S$ and $c_L$ the cost per request of each model, and $c_R$ the cost of routing itself, the expected cost per request is

$$C(p) = c_R + p\,c_S + (1 - p)\,c_L .$$

Quality falls as $p$ rises, because the router has to send harder requests to the small model. Writing $q_L$ for the large model's quality and $\bar{q}_S(p)$ for the small model's average quality on the requests it receives,

$$Q(p) = (1 - p)\,q_L + p\,\bar{q}_S(p).$$

The service should route as much as it can while keeping quality above an agreed floor $Q_{\min}$:

$$p^{\ast} = \max \{\, p : Q(p) \ge Q_{\min} \,\}, \qquad \text{saving} = \frac{c_L - C(p^{\ast})}{c_L}.$$

Illustrative parameters

Cost

Cost falls in a straight line as more requests go to the small model.¹

• With routing
• Large model only

View data as a table

Share routed to the small model	With routing	Large model only
0%	£12.20	£12.00
10%	£11.15	£12.00
20%	£10.10	£12.00
30%	£9.05	£12.00
40%	£8.00	£12.00
50%	£6.95	£12.00
60%	£5.90	£12.00
70%	£4.85	£12.00
80%	£3.80	£12.00
90%	£2.75	£12.00
100%	£1.70	£12.00

Source: Illustrative figures, not measurements.

Quality

Quality holds up while the router is only passing on easy requests, then falls away quickly.

View data as a table

Share routed to the small model	With routing
0%	92%
10%	92%
20%	92%
30%	91%
40%	90%
50%	89%
60%	87%
70%	83%
80%	79%
90%	74%
100%	67%

Source: Illustrative figures, not measurements.

With a 90% floor, the largest share that can be routed is about 40%, where cost is £8.00 per 1,000 requests against £12.00 for the large model alone.

Energy

The same logic applies to energy, where the gap between model sizes is larger still.

View data as a table

Category	Energy per request
Router	0.01 Wh
Small model	0.30 Wh
Large model	3.00 Wh

Source: Illustrative figures, not measurements.

Next steps

1. Measure $c_S$, $c_L$ and $c_R$ on UK-hosted hardware.
2. Build an evaluation set of real, anonymised public-sector requests.
3. Agree $Q_{\min}$ with the services that would use it.²

Footnotes

1. Linear because each request goes to exactly one model. Batching and caching would bend the curve; they are left out of this draft. ↩

2. The right floor will differ by use: a drafting assistant can tolerate more than a service giving guidance to the public. ↩

How to cite

Toby Chen (2027). Right-sizing models: routing requests by complexity. SovereignStrUKture. Available at https://sovuk.vercel.app/research/routing-by-complexity