Which host should serve your open model?

An open model is sold by ten or more hosts within weeks of release. Same weights, but each host chooses its own quantization, price, hardware, cache policy, and defaults, and those choices move month to month. Compound runs your workload against each host, pinned and verified, and gives you one table: success, cost, latency, and cache hits per host.

Read the docs See example reports

Apache-2.0. Runs locally against your own keys. Dry run by default.

$ compound-bench providers

Three steps.

1. See who serves the model

compound-bench providers <model> lists every host on OpenRouter serving it, with quantization, context, price, and whether it is up. Add Doubleword or any OpenAI-compatible host of your own.

2. Run your workload on each

Pick a benchmark (terminal-bench, tau2, BFCL, DS-1000, MMLU) or import your own traces. Each host is pinned with fallbacks off, and the host that actually answered is recorded on every call.

3. Read one table

Success, cost per task, latency, tokens per second, and cache-hit rate per host, with confidence intervals. Every number traces back to a per-call record you can inspect.

Start here.

quickstart 
# OpenRouter key in .env
git clone https://github.com/aktasbatuhan/compound
cd compound && uv sync --extra dev

# who serves this model
compound-bench providers z-ai/glm-5.3-flash

# a dry run: prints the plan, spends nothing. Add --go to execute.
compound-bench run terminal_bench --model z-ai/glm-5.3-flash \
    --providers openrouter/deepinfra/fp8,openrouter/z-ai/fp8 \
    --tasks hello-world --trials 1
  • Nothing is spent until you say so. Every run is a dry run without --go.
  • Bring your own host. A vLLM box or any OpenAI-compatible endpoint sits in the same table as the OpenRouter routes.
  • Your traces, if you have them. Import from Langfuse or JSON and replay real traffic instead of a benchmark. How it works.