All of it, including what got worse.
One fabricated reference deployment — a mid-sized support-chat and document-Q&A product running eight million calls a month — taken from a single frontier tier to a routed ladder over six months. Every figure below comes from that one model, so the numbers can be checked against each other.
Three numbers we would rather not print
They are near the top instead of in a footnote, because a page that only improves is a page nobody should believe.
+1%
P99 latency got 1% worse
Escalated requests make two hops, and the tail is where they land. Median latency fell 58% in the same run; that trade is the product, not an accident.
20%
/review only saved 20%
Its quality floor sits at a judge p10 of 84, close to what the tiers below the frontier can reach. There was never much headroom to take, and taking it anyway would have broken the floor.
11%
Over-routing costs about 11% of the bill
8.7% of calls are scored a rung too high and quietly cost more than they had to. Tightening the thresholds shrinks that and grows the escalation rate instead.
The reference deployment
Fabricated, fixed, and used everywhere on this site. If a figure elsewhere contradicts this table, this table is right and the figure is a bug.
| Parameter | Value | Note |
|---|---|---|
| Monthly calls | 8,000,000 | 3 / 12 · average / peak requests per second |
| Input / output tokens per call | 1,800 / 320 | 960 · a deep-reasoning answer bills more output for the same question |
| Baseline tier, before routing | L-3 | 6% of baseline calls were retries |
| Endpoints | 4 | traffic shares are in the next table |
| Tiers in the ladder matrix | 23 | vendor-neutral codes across seven providers, four of them self-hosted weights |
Four endpoints, each against its own floor
Every endpoint declares its own floor, in its own unit — a judge-score percentile for prose, a schema pass rate for structured output. All four clear the floor they declared. The savings differ by a factor of three, and the reason is always the floor.
Weighted by share this comes to 65.0% before shadow replay is charged back, and 64% after.
Judge score by difficulty band and tier
This is the table that should have been used to pick a model in the first place. Measured on one undifferentiated sample, every tier below the frontier looks bad — because the routine questions and the hard ones are averaged together. Cut the same traffic by difficulty band and the cheap tiers are indistinguishable from the frontier exactly where 42% of the traffic lives.
| Tier | D1 routine · 42% | D2 standard · 34% | D3 complex · 19% | D4 high-stakes · 5% |
|---|---|---|---|---|
| C-0 | 88.2 | 85.0 | — | — |
| S-1 | 86.8 | 79.4 | 66.1 | 51.8 |
| M-2 | 89.1 | 86.1 | 76.3 | 63.2 |
| L-3 | 90.4 | 89.8 | 87.9 | 78.6 |
| L-3R | 90.6 | 90.2 | 89.4 | 85.4 |
What that averages out to
| Basis | Blended judge score |
|---|---|
| Baseline, everything on the frontier tier | 89.1 |
| Routed, first pick only | 85.8 |
| Routed, after the ladder | 86.8 |
The tail went the other way
Blended quality fell 2.3 points and we are not hiding it. The p5 tail improved 2.8 points, because a declared floor forces a redo on the worst answers — something a single-tier setup never even notices, since it has nothing to compare them against.
- +2.8The tail went the other wayp5 68.4 → 71.2
Six months of bills
The line that matters is the gap between what was paid and what the same traffic would have cost on a single frontier tier. Traffic grew 30% over the same six months, which is why the counterfactual climbs while the bill falls.
| Month | Calls (M) | Routed coverage | Bill | If all frontier |
|---|---|---|---|---|
| M0 baseline | 8.0 | 0% | $86.5k | $86.5k |
| M1 | 8.2 | 15% | $80.1k | $88.6k |
| M2 | 8.5 | 35% | $71.3k | $91.9k |
| M3 | 8.9 | 60% | $59.2k | $96.2k |
| M4 | 9.4 | 82% | $48.2k | $101.6k |
| M5 | 9.9 | 95% | $41.8k | $107.0k |
| M6 | 10.4 | 100% | $40.4k | $112.4k |
Three of these five lines are ours, not the providers’: the low-tier calls thrown away during an escalation, the 3% mirrored to shadow, and the amortised classifier. Together they are $0.21 per thousand calls — 5.4% of the routed cost. We print them because leaving them out is how a deck beats reality.
| Line | USD per 1,000 calls |
|---|---|
| model inference (5 tiers) | $3.669 |
| escalation waste | $0.073 |
| shadow replay (3% of traffic) | $0.105 |
| fingerprint classifier | $0.030 |
| judge scoring (6% sampled) | $0.007 |
| Total | $3.884 |
Cost per resolved call
Per-token billing counts attempts. This counts answers someone actually accepted, and puts every retry, discarded low-tier attempt, judge sample and shadow call in the numerator. It is the only cost basis we argue from.
| Basis | Unit cost | Accepted rate | Cost per resolved call |
|---|---|---|---|
| Baseline | $10.81 | 94.1% | $11.49 |
| Routed | $3.88 | 94.6% | $4.11 |
Latency, both distributions
Most requests got faster because most requests were never hard. The ones that escalate pay for two hops, and they are concentrated exactly where the tail already was.
| Percentile | Baseline | Routed | Delta |
|---|---|---|---|
| P50 | 2,265ms | 955ms | −58% |
| P75 | 3,532ms | 1,927ms | −45% |
| P90 | 5,911ms | 4,223ms | −29% |
| P95 | 7,545ms | 7,257ms | −4% |
| P99 | 13,332ms | 13,511ms | +1% |
What Switchyard itself adds
Six milliseconds at the median, eleven at p95, nineteen at p99 — of which the fingerprint is nine at p95 and the decision plus budget check is four. It is a rounding error against the model call it chooses, which is the only reason any of this is worth doing.
- 11msRouter overheadp50 6 · p99 19
- 9 / 4 msfingerprint / decision & budget, p95p95
One brownout, replayed
An illustrative incident: the frontier provider’s 429 rate goes from 0.3% to 31.4% inside eight minutes. What follows is what the state machine did about it.
Timeline
- t = 0
- Upstream 429 rate at 0.3%. Normal. The rolling 60-second window is the only thing watching.
- t = 4 min
- Forty seconds after the window crossed the trigger, /chat and /docqa enter brownout. Detection target is 60 seconds; this took 40.
- Action
- Top rung swapped from the frontier tier to the mid tier, output cap cut from 1,024 to 512 tokens, the reasoning tier disabled.
- Peak
- Upstream failure peaks at 31.4%. User-visible errors peak at 1.1%. At the worst point 41% of answers come from a degraded rung and the blended judge score dips to 81.6.
- t = 30 min
- Upstream back at 0.3%. The cooldown holds the degraded ladder for ten more minutes before the top rung ramps back.
The readout
With 31.4% of upstream calls failing, the user-visible error rate peaked at 1.1%. The price was paid in quality, not availability: 41% of answers came from a degraded rung and the blended score fell about five points for roughly twenty minutes.
Trigger rule: upstream 5xx and 429 above 8% over 60 seconds.
Illustrative incident: frontier provider 429 rate spikes 0.3% → 31.4%