Model routing & fallback layer
CHEAP BY DEFAULT.STRONG ON DOUBT.
live
- −64%unit cost, like-for-like
- 91.9%first-pick pass rate
- 11msrouter overhead, p95
Switchyard is a routing and fallback layer between your application and every model you pay for. It scores each request before it leaves, sends it to the cheapest tier that still clears your quality floor, escalates the ones that don’t, and routes sideways when a provider degrades.
Illustrative model, not measured results.
- 01One request, numbered. It leaves the diagram the moment it reaches its rung.
- 04Escalated. The gate opens and the packet snaps one rung up, still carrying its number.
- Flow speed on a track is that rung’s share of routed traffic. Nothing here is decorative.
02 — why now
THE SAME QUESTION, 43.0× APART.
In eighteen months the count of production-viable general tiers roughly tripled, and the cost of one answer spread out with them. Model selection stopped being a launch-day decision. It is a per-request decision now.
- 9 → 31Production-viable general tiersVendor-neutral codes only, start of the window to now
- 43.0×Spread, cheapest rung to dearestOne question, same input and output length on every rung
- 6Rungs in the default ladderCache, self-hosted, small, mid, frontier, frontier reasoning
03 — the problem
ONE MODEL IS A BET.
Score every tier on one undifferentiated sample of traffic and the answer is always “buy the expensive one.” So you pay frontier prices for the routine half of your calls. The chart is not wrong. It is wrong to measure hard and trivial questions in the same bucket.
A mid-sized AI support & document Q&A product (illustrative)
| Three bills nobody itemises | Value |
|---|---|
| Overprovisioning: routine calls on a frontier tier | $52,300 |
| Retry sink: reruns after failures and truncation | $4,900 |
| Provider wobble: queueing and re-sends during throttling | $2,100 |
| Frontier calls that were actually necessary | $27,200 |
| Total | $86,500 |
Overprovisioning is the large one. Retries and provider wobble are the two that never appear in a model comparison, because a comparison measures one call at a time and these only show up over a month.
And when the model you bet on starts throttling, you have no second choice to fall back to. You have a queue.
04 — difficulty fingerprint
TEN MILLISECONDS. FIVE DIMENSIONS. ONE SCORE.
The fingerprint answers one question, and it answers it before the request leaves your application: how hard is this. Five dimensions, fixed weights, collapsed into one score. No user identity, no history — only this request.
- 5 / 9 / 14 msLatency (p50 / p95 / p99)
- $0.00003Cost per call
- 0.71Agreement with human labels (κ)
- 84.1%Band accuracy, traffic-weighted
Four difficulty bands
| Band | Range | Share of traffic |
|---|---|---|
| D1routine | D < 25 | 42% |
| D2standard | 25–55 | 34% |
| D3complex | 55–80 | 19% |
| D4high-stakes | D ≥ 80 | 5% |
- Size
- 60M encoder, self-hosted
- Fine-tuned on
- 1,200,000 labelled requests
A small self-hosted encoder does the scoring, so working out where a request should go never requires sending it anywhere first.
It gets it wrong sometimes. The calls it scores too low are caught by the ladder. The calls it scores too high are money we wasted, and we print that number rather than round it away. Tighten the thresholds and the waste shrinks while escalation rises. The knob is yours.
- Under-routing
- Scored a rung too low. The ladder catches these — they are the main source of escalation.
- Over-routing
- Scored a rung too high. Nothing catches these: it is money we spent for nothing, roughly 11% of the bill.
05 — route budget
A BUDGET IS A CONSTRAINT, NOT A REPORT.
Each endpoint declares a ceiling in dollars per thousand calls, and the router solves inside it. When headroom runs out the whole ladder shifts down a rung for the rest of the day. Nobody gets an email after the money is already gone.
Every decision is one path through a readable tree. Each request carries the reason it took the edge it took: the band it landed in, the rule that fired, the rung it was given.
Four side rules can override the tree. Strict formats force a schema-validated rung. Fact-sensitive calls lose the smallest rung. Thin budget pushes the whole ladder down. Calls likely to be refused race two rungs and take the first one that clears the floor.
Budget and actual, per endpoint
USD per 1,000 calls
| Endpoint | Quality floor | Budget | Actual | Headroom |
|---|---|---|---|---|
| /chatsupport chat | judge p10 ≥ 74 | $4.20 | $2.81 | 33% |
| /docqadocument Q&A | judge p10 ≥ 78 | $5.50 | $4.22 | 23% |
| /extractstructured extraction | schema pass ≥ 99.0% | $4.80 | $3.68 | 23% |
| /reviewcontract & code review | judge p10 ≥ 84 | $9.60 | $8.65 | 10% |
Budgets are hard caps in this illustrative deployment; every endpoint is running under its own.
06 — fallback ladder
FAIL SIDEWAYS, NOT DOWN.
A ladder is an ordered chain of rungs — a model tier, its parameters, a prompt version, a trigger condition and a maximum wait. A timeout, an error, a schema failure or a score below the floor moves the request to the next rung. The same ladder runs in both directions.
- 91.9%first-pick pass rate8,455 / 9,200
- 8.1%ladder triggered745 / 9,200
- 1.4%Escalation churn< 2.0%
What the ladder costs
The price of a ladder is latency. Median time to an answer falls hard, because most requests never needed a frontier model. The far tail gets slightly worse, because an escalated request makes two hops. That is the honest trade and it is in the chart.
If an endpoint cannot absorb that tail, run two rungs in parallel and pay for the one you throw away.
| Percentile | Baseline | Routed | Delta |
|---|---|---|---|
| P50 | 2,265ms | 955ms | −58% |
| P75 | 3,532ms | 1,927ms | −45% |
| P90 | 5,911ms | 4,223ms | −29% |
| P95 | 7,545ms | 7,257ms | −4% |
| P99 | 13,332ms | 13,511ms | +1% |
07 — quality floor
YOUR FLOOR. NOT OURS.
Each endpoint declares its own minimum: a judge score at the tenth percentile, or a schema pass rate. It is not negotiable. Savings are only ever taken from the headroom above it.
- −2.3Blended judge score89.1 → 86.8
- +2.8The tail went the other wayp5 68.4 → 71.2
The tail went the other way
The blended judge score drops a couple of points and we are not hiding it. The bottom of the distribution improves, because a declared floor forces a redo on the worst answers — something a single-tier setup never notices, since it has nothing to compare them against.
| Basis | Blended judge score |
|---|---|
| Baseline, everything on the frontier tier | 89.1 |
| Routed, first pick only | 85.8 |
| Routed, after the ladder | 86.8 |
| Tier | D1 routine · 42% | D2 standard · 34% | D3 complex · 19% | D4 high-stakes · 5% |
|---|---|---|---|---|
| C-0 | 88.2 | 85.0 | — | — |
| S-1 | 86.8 | 79.4 | 66.1 | 51.8 |
| M-2 | 89.1 | 86.1 | 76.3 | 63.2 |
| L-3 | 90.4 | 89.8 | 87.9 | 78.6 |
| L-3R | 90.6 | 90.2 | 89.4 | 85.4 |
20%
/review · contract & code review
One endpoint in the reference deployment barely saves anything. That is the correct result: its floor already sits close to the ceiling of the tiers that could serve it, so there was never much headroom to take.
08 — cost per resolved call
BILL THE ANSWER, NOT THE TOKEN.
The denominator is answers your product accepted. The numerator includes retries, the lower-rung attempts thrown away during an escalation, judge scoring, classifier amortisation and shadow replay. Per-token billing hides every one of them.
- −64%unit cost, like-for-like$10.81 → $3.88
- −53%Bill$86.5k → $40.4k
- +30%Calls (M)8.0 → 10.4
Three of those lines are ours to pay, and we print them anyway — escalation waste, shadow replay, classifier amortisation. Leaving them out is how a deck beats reality.
The two lines in the chart are the bill that was paid and the bill the same traffic would have cost on one tier. The area between them is the whole argument.
Cost per resolved call
| Basis | Unit cost | Accepted rate | Cost per resolved call |
|---|---|---|---|
| Baseline | $10.81 | 94.1% | $11.49 |
| Routed | $3.88 | 94.6% | $4.11 |
11%
Over-routing
Scored a rung too high. Nothing catches these: it is money we spent for nothing, roughly 11% of the bill.
09 — shadow replay · swap window
NEW TIERS LIVE IN THE SHADOW FIRST.
A slice of live traffic is mirrored to candidate tiers offline. Those responses are never returned to anyone. They only update quality scores and confidence intervals.
- 3%Shadow replay312,000 / mo
- 5,0005,000 shadow samples2–4 d
- ±0.6CI lower bound within 0.5 pointsσ ≈ 22
A candidate clears three gates before it can become a default: enough shadow samples on the same endpoint, enough consecutive days, and an interval whose lower bound stays close to the incumbent. Promotion then ramps in three steps rather than one switch.
Three gates
- 5,000 shadow samples
- On the same endpoint. Not 5,000 across the deployment — endpoints do not generalise to each other.
- Seven consecutive days
- Enough to contain a weekend and a Monday, which look nothing alike.
- CI lower bound within 0.5 points
- The 95% interval’s lower bound may not sit more than half a point below the incumbent.
The sample threshold is not a round number somebody liked. It is the point where the interval narrows enough that half a point of difference stops being noise.
Tier · Status
C-0 reuse layer, H-1 fingerprint / fallback, S-1 D1 primary, S-2 D1 long-context, M-2 D2 primary, M-3 /extract candidate, L-3 D3 primary, L-3R D4 primary (output ≈3×), X-9 removed 2026-06
Tiers leave as well. Last quarter some were promoted, one was rolled back, two were retired. A routing layer that only ever adds is not maintaining a ladder. It is collecting one.
5% → 25% → 100%Then it ramps
10 — brownout · stance
DIM THE LIGHTS, DON’T CUT THE CITY.
When a provider throttles or falls over, the endpoint drops to lower rungs, caps output length and switches its top tier off. It keeps answering. A degraded answer beats an error page.
- 31.4%Upstream failure rateupstream 5xx+429 > 8% over 60s
- 1.1%User-visible error ratedetected in 40s
- 41.2%Answers from a degraded rungBlended judge score 86.9 → 81.6
In the illustrative incident the upstream failure rate goes double digits and the user-visible error rate barely moves. The price is printed next to it: a large share of answers came from a degraded rung, and the blended score sat lower for as long as the incident lasted. Detection is measured in seconds, not in support tickets.
Trigger rule: upstream 5xx and 429 above 8% over 60 seconds.
top rung on /chat and /docqa swapped L-3 → M-2, output cap 1024 → 512, L-3R disabled
Five things we don’t do
- We don't train or host model weights, apart from one 8B classifier/fallback of our own.
- We don't publish leaderboards or named-vendor quality comparisons.
- We don't retain prompt or completion bodies by default — only fingerprints, routing decisions and metrics. Body retention is opt-in.
- We don't set your quality floor. That's a product decision, not a default.
- We don't lock you in. Ladders are plain text; export them and run without us.
Each one costs us something. That is why they are worth writing down.
We will also tell you when not to buy this: one endpoint, one model, a small monthly bill, or a floor that already sits at the ceiling of what could serve it. Where we stand
WE DON’T BUILD MODELS. WE PICK THEM.