Terminal-Bench 4.0 · provider routing · 2026-09-01

Pinned Beats Auto

Letting OpenRouter choose a host for GLM 5.3 Flash made an agent stall almost twice as often as pinning one host, and cost 76% more per prompt token. Five hosts, 1,801 model calls, all arms run at the same time.

Model z-ai/glm-5.3-flash Harness Harbor · terminus-2 Tasks 5 × 2 attempts × 100 turns Calls 1,801 Window 23:36–06:20 UTC

Pinning DeepInfra cut the stall rate from 41.5% to 24.0%.

That gap is z = −5.18, p < 0.0001, and it survives Holm correction for the four arms tested against the router. It is not a close call and it is not one lucky arm out of five.

Stall rate

41.5% → 24.0%

auto vs pinned DeepInfra

Cost / 1M prompt tok

$0.067 → $0.038

auto pays 1.76×

Work completed

3.3×

prompt tokens processed

Hosts auto used

across 299 calls

What was measured

A stall is a call that never finished.

Each arm ran the same five Terminal-Bench 4.0 tasks with the same agent, differing only in which serving host answered the model calls. Four arms pinned a single host through a local proxy; the fifth left OpenRouter to route freely. Every call was recorded at the proxy: which host actually answered, tokens in and out, the cached-token split, billed cost, latency, status.

A stall is a call that returned HTTP 200, streamed keep-alive padding for over 300 seconds, and never delivered a completion. The agent sits idle behind one of these. We saw a single call stall for 1,044 seconds before we capped them, on a host that had answered the previous four calls in 9 to 24 seconds.

Stall rate by arm

Bars show the rate; whiskers are 95% Wilson intervals. Hover for detail.

Green is significantly better than the router, red significantly worse, grey indistinguishable from it: all after Holm correction over four tests. Two arms separate from auto; two do not. Picking a host at random is not the win: picking the right host is.

Cost per million prompt tokens

Billed by OpenRouter, summed over calls that reported usage.

Every figure here is a lower bound. Between 30% and 56% of each arm's calls were abandoned before the response carrying usage arrived: those tokens were billed and we cannot see them. The arms that stall most are understated most, which pushes against the finding rather than for it.

Where the router actually sent traffic

299 auto calls; 150 named a host before finishing.

The router spread one workload across seven hosts and sent the fewest calls to the one that turned out cheapest and most reliable. Every pinned arm stayed on its host: of the 814 pinned calls that returned a provider echo, none was served by the wrong host. The 688 that hung returned no echo.

Every number

Arm	Calls	Stall %	95% CI	Abandoned	$/1M ptok	Cache %	p50 s	p90 s	Hosts
deepinfra	492	24.0	20.4–27.9	147	0.038	69.4	18	131	1
parasail	324	39.8	34.6–45.2	171	0.126	71.8	20	207	1
auto	299	41.5	36.0–47.1	149	0.067	63.8	31	173	7
z-ai	376	42.0	37.1–47.1	186	0.038	88.8	79	232	1
novita	310	51.9	46.4–57.4	174	0.044	85.0	64	249	1

Stall rate vs auto, Holm-corrected over four tests: deepinfra z −5.18 · p < 0.0001 novita z +2.59 · p = 0.010 parasail p = 0.674 z-ai p = 0.886

Trial level

Ten trials per arm. Errored means the trial died before a verdict.

Arm	Trials	Resolved	Errored	Input tok	Cached tok	Output tok
deepinfra	10	1	2	17,022,674	11,653,760	459,375
parasail	10	1	8	5,040,250	3,561,984	828,842
auto	10	0	7	5,329,660	3,319,808	530,215
z-ai	9	0	8	6,881,728	5,936,256	525,060
novita	9	0	9	749,119	514,496	188,919

DeepInfra pushed 3.3× the prompt tokens auto did in the same wall-clock window, and lost 2 trials to errors where auto lost 7. Fewer stalls means more turns actually complete.

What this does not show

Not a quality comparison. One trial resolved in each of two arms and none in the other three. GLM 5.3 Flash does not solve Terminal-Bench 4.0 tasks at any meaningful rate, so this run says nothing about which host produces better answers. It compares reliability, cost and throughput only.
The cache column is not trustworthy. Those are self-reported cached_tokens ratios. In a separate controlled probe, one host reported ~100% cached on a freshly nonced prompt it had never seen. Read cost, which is what we are billed; treat the cache figure as the host's claim about itself.
Costs are lower bounds. 30–56% of calls per arm were abandoned before their usage block arrived. Those tokens were billed and are invisible here.
One model, one night, five tasks. Absolute stall rates moved a lot with congestion , an earlier sequential run of the same setup showed 14.6% for auto in a quiet window and 36% in a busy one. That is exactly why every arm here ran at the same time. The comparison holds; the absolute numbers are a property of this window.
Routing is not stable. An earlier run sent 41 of 41 auto calls to a single host. This one spread across seven. Anyone reproducing it should expect a different mix.

Method

Five Google Cloud VMs, one per arm, all started within the same minute in us-central1-a, each running the benchmark through Harbor against terminal-bench/terminal-bench@4.0.0. Model calls were routed through a local pinning proxy that injects the host selection and records every call. Arms differ in one variable: the provider token.

compound-bench harbor \
  --providers openrouter/deepinfra/fp8 \
  --model z-ai/glm-5.3-flash \
  --tasks html-js-filter,photonic-waveguide-routing,music-harmony,\
bun-sourcemap-leak,foodstuff-beta-activity \
  --attempts 2 --n-concurrent 5 --ak max_turns=100 \
  --ledger-dir artifacts/ledger --go

Stall rates are compared with a two-proportion z-test over calls, corrected across the four arm-vs-router tests with Holm–Bonferroni. Intervals are Wilson score intervals, which behave properly at rates far from 50% with a few hundred observations.