Pick the right provider for your own workload.
The same open model is not the same product on every serving host. Compound runs one model across providers on your tasks, measures success, cost, latency, and cache behavior per host, and gates the switch on a sealed decision set with confidence intervals.
Sweep one model across serving hosts.
The same open weights are not the same product on every host. Name the hosts, pick a
benchmark, add --go. Compound pins each host with fallbacks disabled, verifies
who actually served every call, and reports success, cost, latency, and tokens per second per host.
compound-bench run terminal_bench --model deepseek/deepseek-v4-flash-0731 \ --providers openrouter/deepinfra,openrouter/parasail,openrouter/fireworks,\ doubleword/realtime,doubleword/flex \ --trials 3 --go # drop --go for a free dry run
Sample output: one open model on five pinned hosts, 210 episodes. Hover a row for pass counts and confidence intervals, or read the full report.
Cache is a provider decision too.
Send a model to OpenRouter without pinning and consecutive calls can land on different upstreams, fragmenting your prompt cache. You pay full input price on every call, or eat the latency of a cold host. Pin one provider and prompt-cache reuse stays stable. Compound records the cache-hit rate per host, so the cost you compare is the cost you will actually pay.
Profile every host on six axes.
Every sweep report ships per-host profiles: quality, reliability, speed, determinism, cost, and throughput, from the same episodes and the same billed spend. No pricing-page numbers, only what your workload actually observed.
python -m compound.bench_report artifacts/dsflash # -> report/{summary.json, episodes.csv, per_task.csv, charts.html}
No host wins every axis, and that is the point. In the sample run one host takes speed and reliability but not cost, another takes cost and determinism at a quarter of the speed, a third has the throughput but sheds episodes under rate limits.
The profile you should care about is your workload's, not ours. The chart is a standard artifact of every sweep, so the comparison that matters runs on your tasks, your prompts, your budget caps.
Sample data: all six axes from one terminal-bench run (five hosts, 210 episodes). Pick an axis to see which host wins it, hover a host to focus it, or toggle the legend. The winner changes with the axis, which is the whole point: find yours.
One compound.yaml. One content-addressed cache.
Everything runs locally: SQLite storage, your keys in .env, your traces never leave your machine. Each stage writes evidence the next one reads.
Bring your traces
Langfuse export or portable JSON. Redaction runs before anything is persisted. No eval authoring.
Traces to cases
Provenance-typed eval cases split into immutable train, validation, calibration, and sealed decision partitions.
Replay candidates
Any model x provider x quant replays against your graded corpus from the cache. Re-deciding costs $0.
Close the gap
GEPA evolves a cheaper candidate's prompt on train and val cases, never the sealed set. "No improvement" is a first-class outcome.
Ship on a verdict
The gate decides on the sealed set with a rule declared before anyone looks, and emits one of five honest verdicts.
One command. Five honest verdicts.
Move a support task from Opus 5 to the cheaper kimi-k3? Run the candidate and the reference on the sealed decision set under a rule you declared before looking. The same command produces exactly one of five outcomes, and if the data cannot support a decision it says so instead of rounding noise up to a recommendation.
compound gate support --candidate kimi-k3 --reference opus-5 \ --metric task_success --max-regression 0.02 --reason "quarterly cost review"
What keeps a verdict honest.
Every number is reproducible, every comparison shows its fingerprint, and no run spends a cent you did not authorize.
Money-safe by default
Without --paid, zero provider calls, just an estimate. Paid runs need an enabled budget, a hard USD limit, and a --cap.
Fingerprint caching
A content-addressed key over case, prompt, tool schema, sampling params, provider revision, and judge version: the difference between a $2 and a $200 run.
Sealed decision partition
Optimizers, prompt selection, and judge tuning never see the decision set. Opening it requires a stated --reason.
Calibration-gated judges
A judge feeds a gate only after it out-agrees human labels (Cohen's kappa with a bootstrap CI). Until then it abstains.
Intervals, never bare means
Paired-bootstrap confidence bounds on every comparison. The gate emits a non-inferiority decision, not a point estimate to eyeball.
Provider pinning, verified
One flag isolates a serving host with fallbacks disabled, and the served host is recorded on every call. Works for third-party agent harnesses through a local proxy.
Five benchmarks run today. Adding one is a registry entry.
Each ships with its official grader and a one-time prepare step, so a fresh
clone can run any of them. compound-bench run <name> --tasks retail:10 --go
runs a task subset; drop --go and it is a free dry run.
| Benchmark | What it measures | How it grades |
|---|---|---|
| tau2 | interactive tool-calling support (airline / retail / telecom) | live user simulator + official reward |
| bfcl | single-turn function-call generation | official AST checker |
| ds1000 | data-science code generation | official tests in a pinned container |
| mmlu | multiple-choice knowledge, 57 subjects | exact letter match, no judge |
| terminal_bench | agentic terminal tasks | official harness in Docker |
Wiring a new one is a single registry entry backed by a partitioned manifest: keep it local for your own tasks, or contribute it upstream. See the adapter interface, and the in-progress Harbor / terminal-bench 2.0 adapter.
From production traces to a gated verdict.
# local-first, no account git clone https://github.com/aktasbatuhan/compound cd compound && bun install # traces in, verdict out bun run compound import export.jsonl --importer langfuse bun run compound curate support bun run compound gate support --candidate kimi-k3 \ --reference opus-5 --reason "quarterly cost review"
- Declare the rule first. The gate's threshold is content-hashed before anyone looks at results; the sealed set cannot be reverse-fit.
- Assertions before judges. Deterministic checks are free; judge tokens are not. Cheap filters run first, every time.
- Budget is a hard wall. Paid runs stay off until you enable them and set a USD limit, then bounded by a per-run cap.