Score it. Route it. Catch it.
Three mechanisms decide where a request goes: a fingerprint computed before it leaves your process, a budget enforced while the decision is being made, and a ladder that catches whatever the first pick got wrong. This page is the detail you would need to judge whether that can actually work.
Nothing here is adaptive in the hand-wavy sense. Every threshold on this page is a number you set, versioned with the ladder it belongs to, and printed back to you on every decision.
Step 01 — Difficulty fingerprint
TEN MILLISECONDS. FIVE DIMENSIONS. ONE SCORE.
Before a request leaves your application, a small self-hosted classifier scores it on five axes — reasoning depth, context span, format strictness, factual sensitivity, refusal risk — each on a 0–100 scale. Those five collapse into one difficulty score, D.
D is the router’s only input. Not the user, not their plan, not what they asked yesterday. One request, judged on its own text, so the same question always gets the same treatment and the decision stays explainable after the fact.
D = 0.34R + 0.22C + 0.14F + 0.20K + 0.10X
Weights sum to 1.00. They are fixed per deployment and versioned alongside the ladder.
The five axes
| Axis | What it measures | Weight |
|---|---|---|
| R | reasoning depth | 0.34 |
| C | context span | 0.22 |
| F | format strictness | 0.14 |
| K | factual sensitivity | 0.20 |
| X | refusal risk | 0.10 |
Four difficulty bands
D is cut into four bands. The bands, not the raw score, are what the ladder is written against — a threshold you can read in a config file beats a number nobody can picture.
| Band | Range | Share of traffic |
|---|---|---|
| D1 · routine | D < 25 | 42% |
| D2 · standard | 25–55 | 34% |
| D3 · complex | 55–80 | 19% |
| D4 · high-stakes | D ≥ 80 | 5% |
The classifier itself
It is a 60M-parameter encoder you run yourself. It never calls out, so the fingerprint costs one local inference and the text of the request does not leave your side to be scored.
| Parameter | Value |
|---|---|
| Size | 60M encoder, self-hosted |
| Fine-tuned on | 1,200,000 labelled requests |
| Latency (p50 / p95 / p99) | 5 / 9 / 14 ms |
| Cost per call | $0.00003 |
| Agreement with human labels (κ) | 0.71 |
| Band accuracy, traffic-weighted | 84.1% |
It gets the band wrong about one time in six. That is not a caveat in the footnotes, it is the reason the ladder exists.
- Under-routing
- Scored a rung too low. The ladder catches these — they are the main source of escalation.
- Over-routing
- Scored a rung too high. Nothing catches these: it is money we spent for nothing, roughly 11% of the bill.
Tighten the thresholds and over-routing shrinks while escalation — and the latency tail with it — grows. There is no setting where both go down. The knob is yours, per endpoint.
Step 02 — Route budget
A BUDGET IS A CONSTRAINT, NOT A REPORT.
Each endpoint declares a ceiling in dollars per thousand calls. It is applied while the route is being chosen, not reconciled at the end of the month — when headroom runs low the whole ladder shifts down a rung for the rest of the day rather than overspending and emailing you about it.
That is the whole difference between a budget and a dashboard. A dashboard tells you what happened. This changes what happens.
Budget and actual, per endpoint
| Endpoint | Quality floor | Budget | Actual | Headroom |
|---|---|---|---|---|
| /chat | judge p10 ≥ 74 | $4.20 | $2.81 | 33% |
| /docqa | judge p10 ≥ 78 | $5.50 | $4.22 | 23% |
| /extract | schema pass ≥ 99.0% | $4.80 | $3.68 | 23% |
| /review | judge p10 ≥ 84 | $9.60 | $8.65 | 10% |
Budgets are hard caps in this illustrative deployment; every endpoint is running under its own.
The router is a tree, not a model. Every request carries the path it took and the condition on each edge it crossed, which means a routing decision can be argued with — by you, in a ticket, six weeks later.
Cache hits are counted as a leaf, because they are a routing decision too.
Four side rules
These override the tree. They exist because some failures are not worth being clever about.
- Format strictness at or above 80 forces a rung with schema validation, and a validation failure escalates.
- Factual sensitivity at or above 70 disables the small general tier entirely.
- Below 20% budget headroom the whole ladder shifts down one rung until the end of the day.
- Refusal risk at or above 70 races two rungs in parallel and takes the first answer that clears the floor.
Step 03 — Fallback ladder
FAIL SIDEWAYS, NOT DOWN.
A ladder is an ordered chain of three to five rungs. Each rung is a model tier plus its parameters plus a prompt version, with a trigger condition and a maximum wait. The same ladder runs in both directions: up when the first pick fell short, down when the budget or the upstream says it has to.
Four things move a request to the next rung: a timeout, an upstream error, a schema validation failure, or a judge score under the endpoint’s floor. Nothing else. There is no mood in it.
What moves a request up
- Timeout
- The rung’s max wait elapsed. The in-flight call is abandoned, not waited out.
- Upstream error
- A 5xx or a 429 from the provider, after the retry the rung is allowed.
- Schema failure
- Structured output did not validate against the endpoint’s schema.
- Under the floor
- The sampled judge score came in below the floor this endpoint declared.
First picks hold 91.9% of routed calls. The 8.1% that escalate are mostly under-routed requests being corrected, and the average escalated answer gains 12 judge points by moving up — which is the whole argument for having a ladder instead of a bigger default.
What the ladder costs
Median latency drops 58%, because most requests never needed a frontier model in the first place. P99 gets 1% worse, because an escalated request makes two hops. We print both. If an endpoint cannot absorb that tail, race two rungs in parallel and pay for the second one.
- −58%Percentile P502,265ms → 955ms
- +1%Percentile P9913,332ms → 13,511ms
The ladder is a text file
This is the whole configuration for one endpoint. Export it and it runs somewhere else; there is no state hidden in our console that you would have to re-derive.
Thresholds, waits and the brownout trigger are all in one file, versioned with the prompt versions it names.
endpoint: /chat
quality_floor:
judge_p10: 74
route_budget:
usd_per_1k_calls: 4.20
hard_cap: true
ladder:
- tier: C-0 # semantic cache
when: { cache_cos: ">=0.94", same_prompt_version: true }
- tier: S-1
when: { D: "<25", K: "<70" }
max_wait_ms: 1200
- tier: M-2
when: { D: "25..55", or_escalated_from: S-1 }
max_wait_ms: 2500
- tier: L-3
when: { D: ">=55", or_escalated_from: M-2 }
max_wait_ms: 6000
brownout:
trigger: { upstream_5xx_or_429: ">8% over 60s" }
action: { cap_output_tokens: 512, drop_top_tier: true, keep_serving: true }
cooldown_min: 10Step 04 — Brownout
DIM THE LIGHTS, DON’T CUT THE CITY.
When a provider throttles or falls over, the endpoint does not start returning 5xx. It drops its top rung, caps output length and keeps answering — worse answers, on purpose, out loud.
The name is borrowed from the grid, where dimming a city is the designed response to losing supply and blacking it out is the failure.
Four states
| State | Entered when | What it does |
|---|---|---|
| normal | Upstream errors under the trigger on a rolling 60-second window. | Full ladder, declared budgets, top tier available. |
| watch | The error rate crosses the trigger but the window has not closed yet. | Nothing user-visible changes. The router starts counting and the clock on the detection target starts. |
| brownout | The window closes above the trigger — 5xx and 429 over 8% across 60 seconds. | Top rung dropped, output capped at 512 tokens, the reasoning tier disabled. The endpoint keeps serving. |
| recover | Upstream back under the trigger. | The endpoint holds the degraded ladder for a ten-minute cooldown, then restores the top rung on the same 5% → 25% → 100% ramp used for a swap. |
Cooldown exists because a provider that just came back is the most likely thing to go away again. Flapping between states costs more than staying dim for ten minutes.
Step 05 — Swap window
NEW TIERS LIVE IN THE SHADOW FIRST.
Three percent of live traffic is mirrored to candidate tiers. Those answers are never returned to anyone; they exist only to move a quality score and narrow its confidence interval.
A candidate becomes a default only after it clears three gates. Then it ramps — it does not switch.
Three gates
- 5,000 shadow samples
- On the same endpoint. Not 5,000 across the deployment — endpoints do not generalise to each other.
- Seven consecutive days
- Enough to contain a weekend and a Monday, which look nothing alike.
- CI lower bound within 0.5 points
- The 95% interval’s lower bound may not sit more than half a point below the incumbent.
The 5,000 is not a round number picked for a slide. At a judge-score spread of about 22 points, that is where the 95% interval narrows to roughly ±0.6 — which is what makes a −0.5 rule mean anything at all.
Then it ramps
Promotion goes 5% → 25% → 100%, one endpoint at a time, with a rollback switch at every step. Last quarter, in this illustrative deployment: three promoted, one rolled back, two retired.