The same open model, eight hosts: identical quality can cost 9.6x more. Compound is how you find the provider your workload can afford.

Setup

One model, deepseek/deepseek-v4-flash-0731, served eight ways: Doubleword (realtime and flex tiers, direct API) and six OpenRouter routes pinned with fallbacks disabled (DeepInfra fp4, Parasail fp8, Fireworks, Cloudflare, Together, and one arm left on OpenRouter's own unpinned auto routing). The served host is verified on every call, so a pinned run is checked, not trusted.

Every cell shares the same conditions: each task's agent time limit multiplied by three and labeled non-official in every run's metadata, the reasoning mode pinned per host, and the best prompt cache each host offers switched on (implicit prefix caching on OpenRouter, explicit cache markers on Doubleword). Six hosts are fully crossed on both reasoning modes. Together serves only its reasoning-off cell because its reasoning-on path fails a capability probe (below), and the auto arm runs reasoning-on only.

Hosts silently disagree about how your model runs

Before pinning anything, we sent byte-identical requests with no reasoning parameter, and the serving layer made opposite choices: both Doubleword tiers ran the model with reasoning off (0 reasoning tokens across 1,370 calls) while every OpenRouter upstream ran it with reasoning on. Same weights, same request, different model behavior, and nothing in the response flags it. Each host also has its own pin dialect (reasoning_effort versus a reasoning block), and one host even flipped its default between streaming and non-streaming calls. A latency or quality chart comparing hosts that silently disagree on mode is comparing completion lengths, not serving. So we pinned the mode explicitly in each host's own dialect and verified it in the traces; every comparison below is same-mode.

The fair matrix

The result is a single, condition-matched matrix: 8 hosts, 2 reasoning modes, tripled clock, best-available cache, 588 terminal-bench episodes across 14 filled cells, every cell grepped for billing errors before it entered the table. The bars below are reasoning-on resolve rate with the reasoning-off rate behind them for the six fully-crossed hosts; the two partial arms, Together (off only) and unpinned auto (on only), sit in the table underneath.

Fig 1. Six fully-crossed hosts, 42 episodes per cell. Together (reasoning-off only, 20/42) and unpinned auto (reasoning-on only, 23/42) are in the table. Toggle the resolve rate between all runs and wall-clock failures excluded; hover a row for counts, cache share, and the confidence interval.

Table 1. All measured cells at tripled, non-official limits. Cost is the whole 42-episode cell. OpenRouter cost is the provider-billed amount measured per call; Doubleword returns no billing API, so its cost is derived from its own reported token meters at its published rates. Cache share is measured from per-call cache-token meters. The toggle above recomputes the resolve column two ways: over all runs, or with wall-clock-bound failures (episodes that spent the whole tripled limit on API time without finishing) removed from the denominator, which isolates model quality from decode speed.

Reading the matrix as a budget

A resolve rate does not pay an invoice. The reason to measure cost, decode speed, and cache is that each one becomes a line on your bill, and which line dominates depends entirely on the shape and volume of your workload. The same matrix points at three different providers depending on what you run. These figures scale our measured 42-task cell to a hypothetical 100,000 task-runs a month so the gap is legible as money, not as a decimal.

The point is not that one of these hosts is best. It is that your workload picks a different one, and the only way to know which is to run your own numbers. The rest of this page is the evidence behind each axis; the next section shows how little it takes to reproduce on a benchmark or on your own traces.

1. The ability is fixed, so the choice is economic.

With the clock unbound, doubleword-realtime with reasoning off resolves 22/42 (52.4%, Wilson 95% CI [37.7, 66.6]), statistically identical to the five OpenRouter hosts with reasoning off pooled at 104/210 (49.5%, CI [42.8, 56.2]), Fisher exact p = 0.87. The result survives the discriminating-task sensitivity check (p = 0.82), so it is not an artifact of the easy tasks everyone passes. None of doubleword-realtime's 42 reasoning-off episodes were clock-bound at all, so this is a clean read: the same weights reach the same resolve rate once time pressure is off the table. This matters because it fixes the variable you were worried about. The model's ability does not depend on the host, which means every difference in the rest of this report, cost, decode speed, cache, whether reasoning fits the clock, is the provider's economics, not the model's intelligence. Economics is exactly what you get to choose.

2. Thinking does not fit slow decode, even at 3x.

Pool reasoning-on quality and the split is real: the two Doubleword tiers together resolve 34/84 (40.5%) against the four fast OpenRouter hosts' 99/168 (58.9%), Fisher exact p = 0.0073. The mechanism is not model quality, it is the clock. Doubleword-realtime with reasoning on still spent at least 85% of the tripled limit on API time in 20 of its 24 failures; flex is worse at 25 of 26. At roughly 20 tokens/s, thinking does not fit even a window three times the official one. Reasoning-on slow decode is a throughput ceiling, not a tight-deadline artifact you can buy your way out of with more wall time.

3. Reasoning pays only where decode affords it.

On the four fast hosts, reasoning on resolves 58.9% (99/168) against reasoning off's 50.0% (84/168) on the same four hosts: +8.9 points, p = 0.125 across the full corpus and p = 0.056 on the ten discriminating tasks (+12.5 points there). Reasoning adds roughly nine to thirteen points on fast hosts, but the effect does not clear significance at this sample size, so we report it as directional rather than established. The direction is consistent with finding 2: reasoning is worth its tokens only on a host whose decode leaves room to spend them, and the hosts that clear 76 tokens/s have that room while the 20 tokens/s tiers do not.

A trap worth naming, since Fig 1 lets you toggle it: exclude every wall-clock failure and Doubleword's reasoning-on rate leaps to 87% pooled (34/39), above the fast hosts. That is survivorship, not a quality edge. Excluding clock-bound runs removes Doubleword's attempts on exactly the long, hard tasks it never finished, leaving mostly the easy tasks everyone solves. The honest test is task by task, clock-bound runs dropped from both sides: against the four fast hosts, Doubleword wins 2 tasks, loses 2, and ties 7 with reasoning on. In reasoning off, where the clock is not a factor at all, the two are dead even, 47.6% versus 50.0% pooled and 4 wins, 4 losses, 6 ties task by task. Doubleword never beats the cloud routes on quality; it matches them. Its lower reasoning-on score is the clock, not worse answers. Choose Doubleword for data control and price, and only where its decode speed leaves reasoning affordable, not for a quality edge that the matched comparison does not show.

4. Six of thirty candidate endpoints could not serve the workload.

Eight hosts made the matrix; more were probed and rejected with receipts, because a route that cannot serve the real workload should fail the probe, not quietly degrade a run. Baseten and CoreWeave refuse response_format under require_parameters, so a schema-constrained agent call never lands. The deepseek-origin route advertises schema support in its metadata but the router 404s the same call. Together silently disables thinking under a json_schema request: across three reproducible probes it returned a different wrong arithmetic answer each time while Cloudflare answered 198849 correctly with 56 reasoning tokens, which is why Together appears reasoning-off only. Each exclusion is a deterministic probe repeated three times. In total 6 of 30 listed endpoints could not serve this workload as specified, and none of that shows up on a model card.

5. Cache is a serving decision, and hosts differ more than 2x.

Cache economics dwarf rate cards for agentic workloads, where every turn resends a growing transcript. A growing-prefix replay of a real 41-message transcript, 12 turns per session, three sessions per route, measured steady-state hit ratios (cached tokens over prompt tokens on turns 2 to 11) that span from 59.8% to 82.3% across working caches: Doubleword marked and Together both 82.3%, DeepInfra 75.3%, Fireworks 71.5%, Parasail 69.0%, Cloudflare 63.8%, and unpinned auto 59.8% as the weakest working cache, and a Doubleword stock client at 0.0%, the opt-in trap that a default client walks straight into. The fair-tier cells in Table 1 show the same story on the real workload: cache share runs from 55% on Cloudflare-on to 95% on Doubleword-flex-off. The cheapest rate card does not win if the client never warms its cache.

6. Unpinned routing holds quality, but its cost verdict is conditional.

The auto arm, left on OpenRouter's default routing, resolves 23/42 (54.8%), a statistical tie with the pinned hosts. Cost is where it turns, and the direction depends entirely on the workload. On our schema-constrained agent traffic auto billed $1.16 against DeepInfra's $0.43 for the same resolve rate, 2.7x more, with 62% cache share versus 82%, because the schema requirement excludes the cheap long-tail upstreams and auto then concentrates on premium upstreams (its top pick took 63% of its calls). But on the small, schema-free replay calls the sign flips: auto averaged $0.0041 per session against pinned DeepInfra's $0.0046, because there the cheap upstreams are admissible and their list price outweighs the cache loss. Both are measured. The honest claim is that auto fragments cache and makes cost unpredictable; whether it costs more or less depends on whether your workload's constraints exclude the cheap upstreams.

The frontier below plots all fourteen cells against what each costs to run. Toggle the horizontal axis between dollars, total tokens, and wall-clock hours, and the Pareto-best set reshuffles: the shaded step marks the non-dominated cells for that axis, and dominated cells fade back.

Fig 2. Every point is one full cell at tripled, non-official limits, so the cost, token, and hour figures all cover the whole cell. Doubleword cost is derived from its own token meters at published rates; OpenRouter cost is the provider-billed amount measured per call. Frontier cells are drawn full-strength, dominated cells at 40%.

7. Four outcomes, kept separate: pass, timeout, error, fail.

The single most important choice in this report is that a timeout is not a failure. A run that spent the whole tripled clock still working is a fact about the host's speed, not the model's ability, and an ungraded harness error is not a verdict at all. So we score every trial into four buckets and never fold them together: pass, fail (wrong on merit), timeout, and error. The grid below is that scoring, one square per trial. It is also where the corpus shows what it can and cannot discriminate. Not every task carries signal. Three of the fourteen tasks are never passed by any host (git-multibranch, chess-best-move, and cron-broken-network, all also near-zero at official limits, so not a 3x artifact; cron-broken-network in particular fails as an ungraded terminal-bench parse error rather than a model loss, symmetric across every host) and one is solved by everyone (fix-permissions), with a second missed only once (crack-7z-hash.easy, 38/39). That leaves ten tasks doing the discriminating, so we ran every quality comparison twice, on the full corpus and on the discriminating ten, and report only claims that agree in direction between them. With 42 episodes per cell, within-mode spreads between hosts are not significant and we report them as ties: the widest on-mode gap, 61.9% to 57.1%, is p = 0.82. The grid below unpools every one of the 588 episodes into a single square, so both the discrimination and the amber wall of timeouts in the slow-decode reasoning-on Doubleword columns read at once.

Fig 3. Every trial as its own square, four outcomes kept separate on purpose: pass, fail (the model was served cleanly and got it wrong), timeout (it was still working when the tripled clock expired), and error (ungraded: a harness parse failure or a model output-token overflow, never a quality verdict). Tasks are ordered hardest first by pass count; hover any square for its task, arm, trial, and outcome. This is the whole argument in one picture: the amber in the reasoning-on Doubleword columns is timeout, not fail. The same tasks the fast hosts pass in black, Doubleword would pass too if its ~20 tokens/s decode fit the clock. Read the amber as latency, not as a dumber model.

Anatomy of the failures: model, clock, or infra

A resolve rate is only as honest as the failures behind it. We audited every one of the roughly 13,300 model calls in the matrix, reading each task's grader verdict against its per-call traces, and sorted every non-resolve by cause: the model was served cleanly and got the task wrong, or the model was still working when the tripled clock expired. No retained host produced a serving error. Three real runs, one clean success and one of each failure cause, taken straight from the traces:

Across the fleet the split was clean: every non-resolve on a retained host was either a model-quality miss or a disclosed clock-bound timeout, and not one of the roughly 13,300 audited calls on the retained hosts returned a serving error. One task, cron-broken-network, fails as an ungraded terminal-bench parse error on every host and is kept in the corpus but treated as unscored, not as a model loss.

See three runs, turn by turn

The grid above is 588 squares. Here are three of them opened up: real terminal-bench runs replayed one turn at a time, showing the terminal the agent saw, what it reasoned, and the command it typed. One passes, one is wrong on merit, and one runs out the clock on the very same task a faster host finishes. Pick a tab.

Real traces from the run, trimmed to the turns shown; the full per-episode logs (prompt, model response, terminal recording) ship with every sweep. The fail and timeout tabs are the identical task on a fast versus a ~20 tokens/s host: same 5-of-6 partial progress, but on the slow host the clock expired mid-work, which the grid scores amber, not grey.

Serving metrics: first token, decode, reliability

Quality is one axis; how a host streams is another, and it is a separate experiment. A streamed harness replayed three real agent payloads (2K, 8K, 25K prompt tokens) against the five original hosts in both modes, every hour through a night (rounds 1 to 11, roughly 600 analysis calls plus smoke), measuring time-to-first-token and decode speed from chunk timestamps with prompt caches deliberately busted. The four realtime hosts tie at 0.9 to 1.3s median time-to-first-token, while the flex queue tier answers in one flush (8s median without reasoning, 62s with). Decode speed is the real spread: roughly 108 tokens/s on Fireworks, 95 on DeepInfra, 76 on Parasail, and 20 on Doubleword realtime, and the pools differ in steadiness too, with the big shared pools holding within about ±10% all night while Doubleword swung between 8 and 45 tokens/s. Reliability is its own axis: Fireworks was the only host to error at all, 21% of its overnight calls hit shared-pool 429s, and the daytime rounds we kept running erred at a similar rate. Congestion on a shared pool is a lottery, and pinning buys attribution, not immunity. These serving numbers cover the five original hosts only; the four hosts added for the fair matrix were not in the timing harness.

What this says about picking a provider

The questions that decided these rows never appear on a pricing page: what mode does the host actually run your model in, how fast does it decode when the model thinks, does your agent's task budget survive that speed, can the host even accept your schema-constrained calls, does the pool shed your traffic under load, and does your prompt cache stay warm. Every one of them is measurable on your own workload before you commit, which is the point of Compound: fix the model, pin the hosts and the mode, sweep, and read one table of quality, latency, and cost per route.

Statistical notes

• Intervals are Wilson 95% binomial CIs on 42 episodes per cell (14 tasks × 3 trials).
• Every quality comparison was run twice, on the full 14-task corpus and on the ten discriminating tasks, and only direction-stable claims are reported. Parity: p = 0.87 full, p = 0.82 discriminating. Reasoning value on fast hosts: p = 0.125 full, p = 0.056 discriminating, reported as directional rather than significant. Thinking-versus-slow-decode split: Fisher exact p = 0.0073 full, p = 0.0001 discriminating.
• Within-mode host spreads are not significant at this n and are reported as ties (widest on-mode gap p = 0.82). We do not use winner language for any single-host quality gap.
• The clock-exhaustion counts are per-episode joins of traced API time against each task's own tripled max_agent_timeout_sec, not estimates. "Burned ≥85% of the limit" counts API time only and excludes container and tool time, so it understates true clock pressure.
• OpenRouter costs and cache shares are provider-measured per call from the traces. Doubleword returns no cost field and has no billing API, so its cost is derived from its own reported token meters at its published rates; its cache share comes from measured cache-token meters.
• Serving metrics come from streamed responses on the five original hosts: time-to-first-token and decode rate measured from chunk timestamps, prompt caches busted with a nonce, each host called with the same 2-way concurrency the benchmark used.
• Every cell was grepped for billing-error episodes before entering the matrix. Episode pass/fail comes from each task's own test suite; there is no judge anywhere in the loop.

Run it on your own numbers

This report is one benchmark. The point of Compound is that the same measurement runs on whatever you care about: a public benchmark like terminal-bench, or your own production traffic imported from Langfuse and replayed against every provider from cache. Any model, your pick of hosts, dry run by default:

# clone and install the benchmark engine
$ git clone https://github.com/aktasbatuhan/compound
$ cd compound && uv sync --extra dev

# discover pinnable hosts for a model
$ compound-bench providers deepseek/deepseek-v4-flash-0731

# run the sweep (drop --go for a free dry run)
$ compound-bench run terminal_bench \
    --model deepseek/deepseek-v4-flash-0731 \
    --providers openrouter/deepinfra,doubleword/realtime,doubleword/flex \
    --tasks fix-permissions,create-bucket --trials 3 --go

Reasoning mode pins with COMPOUND_REASONING=on|off, which Compound's pinning proxy translates into each host's own dialect. The proxy also pins with require_parameters and verifies the served host on every call, so a capability gap or a silent reroute fails fast instead of contaminating a run.