Benchwarmers
Benchwarmers is an MCP panel gateway: you can discover collections, ask a collection for multiple model answers, and judge those answers.
List collections to see what each is for, its members, explorer seats, reply mode, and shared constraints.
Ask a collection with a prompt to get every seat’s answer side by side, optionally with a system message, extra explorer models, or a merged answer.
Record a verdict per seat using its call id, marking whether it was useful and listing true or false claims to feed routing and model records.
Check pending verdicts for a Claude Code session/transcript, mainly for the plugin’s Stop hook.
Supports the Gemini API format, allowing panel requests to Gemini models through Google's API.
Supports the Ollama API format, enabling panel requests against local or Ollama-served models.
Supports OpenAI chat completions and Responses API formats, enabling one request to fan out to a panel of models and return side-by-side answers, including through OpenAI-compatible providers or hubs.
Benchwarmers is a self-hosted panel gateway for LLMs. Your app asks for one of your collections, and the whole panel answers: the models you picked for that kind of work, plus explorer seats that rotate in the models you have used least. Every answer comes back side by side, labelled by model and seat. You check them against the real work and record a verdict, and those verdicts decide who sits on the next panel.
What it is best at:
Asking several models at once. One request, in the OpenAI, Anthropic, Gemini or Ollama format, a panel of answers. Where they disagree is where to look.
Agents too. Claude Code and other tool-calling clients use a collection through route mode: one member per turn, kept for the conversation, with failover, and every call on the record.
Reviewing. Code review, claims about code, designs: different models catch different things, and two models agreeing is not proof.
Playing to each model's strengths. A collection per kind of work (reviewers, designers, enumerators), each with the models that do it best.
Giving every model playing time. Explorers cycle through the rest of the bench, so new and free models get tried beside the ones you trust, without you choosing them by hand.
Benchmarking on your own projects. A review suite seeded from your own repositories, plus your verdicts on real work, instead of public leaderboards.
Knowing who to bench. Who is confidently wrong, who has stopped answering (retired automatically), and who is resting after a rate limit.
Keeping free and cheap models useful. Health probes, rate limits taken from what providers say and enforced, and one endpoint for every app, over any OpenAI-compatible provider or hub, or Anthropic's and Gemini's own APIs.
Not another gateway. Gateways such as OmniRoute, freellmapi, LiteLLM and OpenRouter choose where a request goes by availability, quota, cost and speed. Benchwarmers finds out which of your models is right, and ranks them by that. It sits happily on top of one: the gateway keeps the accounts and quotas, Benchwarmers keeps the evidence. COMPARISON.md has the side-by-side tables and the set-up.
Related tools | Examples | What they do | What Benchwarmers does differently |
Gateways and routers | OmniRoute, freellmapi, LiteLLM, OpenRouter, Portkey | One request to one model, picked by quota, cost, speed and health, with fallback | Fans one prompt out to a panel and returns every answer; for agents, routes to one member by its verdicts. Runs on top of them |
Councils and ensembles | OpenRouter Fusion, OmniRoute's fusion, LLM Council, Mixture-of-Agents | Several models, then a judge or chairman writes one answer | Keeps every answer, because who said what is the evidence; one merged summary only if you ask |
Side-by-side chat | Open WebUI, LMArena | Several answers shown together, thumbs up or down for a leaderboard | Records a verdict per answer, with its false claims, from you or the calling agent, and routes by them |
Evaluation harnesses | promptfoo, Inspect, SWE-bench | Score models offline on fixed test sets | Scores them on your real work and your own code, and keeps rotating new models in |
Contents
Day-to-day use (the dashboard's pages, connecting apps, onboarding in bulk) is in USAGE.md. Running it on a server or a hosting platform is in HOSTING.md. A coding agent setting it up for someone should start with AGENTS.md.
Related MCP server: llm-gateway-mcp
Try it
Version 1.1. It runs with Python 3.11 or later on Windows, macOS and Linux, from PyPI, npm, a checkout or Docker. Report rough edges as issues, and read the code where these pages are thin.
Quickest, with Python 3.11+ installed, from PyPI:
pipx install benchwarmers # or: uv tool install benchwarmers
benchwarmers # or try it first, without installing: uvx benchwarmersor from npm, with Node 18+ as well:
npx @wendellresearch-ai/benchwarmersEither way you get the same two commands, benchwarmers and its short name
bench. It keeps your data in your user profile (BENCHWARMERS_HOME to
choose another folder), sets it up on the first run, and opens the dashboard
on http://localhost:8122, where steps 3 to 5 below continue. The built-in
model client makes every call, so nothing else needs installing. No
dependencies either way: the app is the Python standard library.
From source:
Get the code and a Python 3.11+ environment:
git clone https://github.com/WendellResearch-AI/benchwarmers cd benchwarmers python -m venv .venv # nothing to pip install: the standard library is enoughThe built-in model client (
metrics/client.py) makes every call. ThellmCLI is an alternative for its plugins: install it in the venv and choose it on Settings (orBENCH_CLIENT=llm, which wins).Make a data folder. Everything that is yours (endpoints, model registrations, collections, verdicts, probe history, task answers) lives there, never in the code:
python scripts/init.py # or: python scripts/init.py <folder>It starts the folder from
examples/(in your OS's per-user data folder unless you name one) and points the checkout at it with a.benchwarmers-homefile;BENCHWARMERS_HOMEoverrides that. With--endpoint NAME BASE_URL --keyit also adds a first endpoint and asks for its key, which goes to the key file in llm's user folder (keys.json, shared with thellmCLI if you have it, not needed otherwise), never into the data folder.BENCHWARMERS_KEY_<NAME>in the environment works too. To move an install to another machine, Settings → Export and import takes the folder as one .zip (settings only, or everything; the endpoints' keys only if you tick them) and imports it on the other install, saving what it replaces tobackups/first.Start the dashboard. With no endpoint yet it opens on adding one: sign in with a provider on its own page (OpenRouter, NanoGPT, Hugging Face, Pollinations, your ChatGPT plan, GitHub Models, or a local Ollama), or paste an API key for any OpenAI-compatible API, a hub such as Open WebUI or LiteLLM included. Anthropic and Gemini keys can also be called through those providers' own APIs (the API choice in the same dialog). Only providers that publish a sign-in for other apps are offered; adding one is an entry in
metrics/connect.py(its docstring says how). A model server already running on this machine, at its usual port or any other its process is listening on, is offered under "Found on this machine", its address and kind filled in: Ollama, LM Studio, Jan, GPT4All, llama.cpp, llamafile, LocalAI, vLLM, Strata and Project Maya; AstrLink, opencodex, ThinkWatch Lite, Claude Code Router, Bifrost and 9router as gateways; Open WebUI, LiteLLM, CLI Proxy API and OmniRoute as hubs, which the bench can add a model to; freellmapi, one-api and new-api as hubs it reads (their quota, and one-api's and new-api's channels). Routers and combos (OmniRoute's combos andauto, freellmapi'sauto,auto:<profile>andfusion) are kept out of onboarding, since a verdict is about one model.python serve.py # http://localhost:8122Onboard a few models from it ("Onboard a model"): tick one or several; each is registered, probed and run through the suite, one at a time, with the progress beside the button. Quick probes now and leaves the suite to the scheduled passes; + ad hoc (with Full or Quick) keeps the bench from ever calling the model on its own afterwards, for a metered or trial key. Tick Vision model for one that reads images and little else: it sits only for questions about images, and is tested on the vision tasks, not the code ones.
Make a collection on the Collections page (the models a panel always includes, and what it is for). The dashboard answers the gateway's routes on its own port (
http://127.0.0.1:8122/v1);python gateway.pyruns the gateway alone onhttp://127.0.0.1:8123/v1.
How it works
Collections and seats. A collection names the models a panel always includes and the shape of work it is for (review, design, enumeration...). Each call adds explorer seats chosen by how long a model has gone unused, so the bench keeps learning about models a router would stop asking. A member that is out of quota or rate-limited keeps its place while an explorer stands in for it until it is back. A collection can set its own calls per request and explorers, and one marked ad hoc (an expensive one, say) runs only when a request names it.
Verdicts. Each seat's answer carries a call id. Once you have checked an answer against the real work, record whether it was useful, and which of its claims were real or false. Verdicts, not vibes, decide the next panel; a model that is often confidently wrong shows as that.
The catalog. Context window, output limit, tools, vision and price come from models.dev for the provider that actually serves a model, then OpenRouter's public catalog and FreeLLMAPI, and anything you set by hand.
Health. Every morning each model gets a one-word prompt; once a week, a short and a long one for speed. A model that stops answering, or that its endpoint no longer serves, is retired automatically after one confirming call, with the evidence kept, and can be restored in one click. A model that is only out of quota until a stated reset is not counted against. The built-in scheduler runs all of this once Settings switches it on; a fresh install spends no calls until then.
Benchmarks. One page of reports: bugs found against false alarms, who is confidently wrong, real work that helped, each model against its own earlier scores (Trend), whether each provider was there when the bench called it (Uptime: busy, unreachable and timed out count as down; your own quota and pacing are left out), and speed. Each report shares as an image or its table. BENCHMARKS.md says how to read them.
Decision models. A model that answers typed questions with a probability (Mercury Decide, say) labels a request's shape, picks a collection for a request that names none (when it is sure, and the reply says so), and classes an error no rule knows. Each is measured daily on questions the bench's own data answers, and each use asks the one that does best.
Rate limits and usage. Limits come from providers' rate-limit headers and error messages, and from what you set per connection or model (requests a minute, day or month; tokens a minute or day). They are kept per connection, since models behind one key share them, and enforced: a call waits when the wait fits, or is refused before it goes out. Every call lands in a monthly usage ledger with its tokens. Quota management has a tile per provider showing what is left, when it resets and who is resting. OpenRouter, xKiro, Requesty, the sign-ins in CLI Proxy API or OmniRoute, and what freellmapi, one-api and new-api say is left are read live, at no cost to quota. Nothing is taken from a provider's docs; NVIDIA's own stated figures are read from its site and watched for changes.
Discover. FreeLLMAPI's free catalog, with models.dev for the details and paid models it lacks, merged into one list of models worth adding, smartest first, each with the routes this install has to it and links to its pages on both. Each Open WebUI connection's own model list adds what a provider serves before either catalog lists it; Refresh lists fetches them all now rather than at the daily fetch.
Asking a panel
Any OpenAI-compatible client works: ask for the model bench/<collection>.
curl http://127.0.0.1:8122/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "bench/<collection>", "messages": [{"role": "user", "content": "Where can this function return a stale value? ..."}]}'The reply carries every seat's answer in the message, and the same seats as
data in a bench field: model, seat, status, seconds and a call id. It does
not wait for the slowest seat: once two have answered, the rest get 60
seconds, and one still answering is named late and recorded when it
finishes. For a chat
window that shows one answer, ask for bench/<collection>:merged: a merger
model writes one answer saying where the seats agreed and disagreed, and every
seat is still in bench. Record a verdict per call once you have checked it:
curl http://127.0.0.1:8122/bench/verdict -H "Content-Type: application/json" \
-d '{"call": "<call id>", "useful": true, "note": "found the real bug", "real": ["..."], "false": ["..."]}'or from the dashboard. See GATEWAY.md for the full contract: images, prompt sizes, who may call, and the reply's shape.
Other clients. The same panel answers in other APIs' shapes, so a client that speaks one of them works with the server's address as its base URL:
Format | Route | For |
OpenAI chat completions |
| OpenAI SDKs, most chat apps |
OpenAI Responses |
|
|
OpenAI legacy completions |
| tools that send a |
Anthropic Messages |
| Anthropic SDKs with |
Ollama |
| Open WebUI's Ollama connection, Ollama apps |
Gemini |
| Google GenAI SDKs with |
curl http://127.0.0.1:8122/v1/messages -H "Content-Type: application/json" \
-d '{"model": "bench/<collection>", "max_tokens": 1024, "messages": [{"role": "user", "content": "..."}]}'Each streams when asked: the reply starts at once, and each seat's answer
arrives as that seat finishes. A panel answers with text, so a request
carrying tools is refused with a 400 that points to route mode
(bench/<collection>:route, one member with its tools; see
below). GATEWAY.md, "Other API formats",
has a curl for each and what is refused or ignored.
From the command line
bench (or benchwarmers, the same program) does what the dashboard does,
without it: ask and judge, and run the bench itself. It is a client of a
running server, so the same commands work against one on another machine.
bench status what is running, and where
bench restart start it again on the code now on disk
bench collections what you can ask
bench ask core "Where can this return a stale value?" -f cache.py
git diff | bench ask core "Review this diff" a pipe is the prompt
bench verdict <call id> --useful --real "the race in refresh()" --false "the SQL injection"
bench models --find mercury the roster, with each model's status
bench candidates --new what your endpoints serve that is not on the bench
bench onboard <served name> --wait onboard it (registered, probed, scored)
bench model retire <alias> --reason "..." and back: bench model restore <alias>
bench collection save reviewers --members a,b,c --description "adversarial review"
bench collection save reviewers --add d --remove a
bench collection save heavy-hitters --ad-hoc --max-calls 4 --max-explorers 0
bench endpoint add openrouter --base https://openrouter.ai/api/v1 (asks for the key, hidden)
bench key create my-app --collections core --rpm 30
bench settings scheduler run --off perf--help on any command lists its options with examples, and --json prints
what the server answered, for scripts. --url points one command at another
server. ask prints every seat's answer on stdout and the table of call ids to
judge them by on stderr, so bench ask core "..." > review.md keeps only the
answers. Retiring a model, deleting a collection or an endpoint and revoking a
key ask first; --yes answers for a script. A key is never an argument (others
on the machine can read those): endpoint add prompts for it, or reads it
from stdin with --key-stdin.
Another program called bench (the Frappe framework's) may come first on your
PATH; type benchwarmers instead, which takes the same commands.
Variable | Meaning |
| the server, default |
| the server's admin token, where it has one; sent as |
| a gateway key, for |
| seconds to wait for a panel, default 900 |
Exit codes: 0 done, 1 refused or failed (the reason on stderr), 2 bad usage, 3
no server at BENCHWARMERS_URL.
From Claude Code and other agents
A running Benchwarmers is an MCP server at /mcp on its own port
(http://127.0.0.1:8122/mcp; /v1/mcp works too), over streamable HTTP,
with four tools: list_collections, ask (a collection, a prompt,
optionally merged, with and system), verdict and pending_verdicts
(the seats still unjudged). Nothing to start per
client: every agent session shares the one server, behind the same Host,
Origin and gateway-key checks as /v1. The tool descriptions carry the
rules: ask a collection, not a favourite model; verify before acting; name
every seat; send a verdict per seat you judged.
Claude Code plugin. The plugin connects to that /mcp and adds a
/benchwarmers:ask <collection> <what to ask> command and a benchwarmers
skill saying when a panel helps (adversarial review, exhaustive enumeration
over real files) and how to judge it:
/plugin marketplace add WendellResearch-AI/benchwarmers
/plugin install benchwarmers@benchwarmersIt finds the server through BENCHWARMERS_URL (default
http://127.0.0.1:8122; set it if you changed BENCH_PORT) and sends
BENCHWARMERS_KEY where keys are required. It also adds a Stop hook: at
the end of a turn it asks the server (the pending_verdicts tool, so nothing
extra runs on your machine) which seats of that session's panels have no
verdict yet, and reminds Claude of them once, as hook feedback. The server
finds the session's panels in its transcript, so the reminder needs a server
on the same machine; with one elsewhere the hook stays silent. A verdict sent
any way counts, the dashboard's included. With the plugin enabled and no
Benchwarmers running, the hook shows a hook-error notice at the end of each
turn, as /mcp shows the server as failed; start the server, or disable the
plugin for that session.
Cursor and Antigravity plugins. The same skill, with the stdio server below (so they need Node and Python 3.11 or later): Cursor's is on cursor.directory, and Antigravity's installs from the repository:
agy plugin install https://github.com/WendellResearch-AI/benchwarmersAny other MCP client that speaks HTTP:
{"mcpServers": {"benchwarmers": {"type": "http", "url": "http://127.0.0.1:8122/mcp"}}}with "headers": {"Authorization": "Bearer <gateway key>"} where keys are
required. A client that only speaks stdio: run benchwarmers mcp once
installed, uvx benchwarmers mcp or
npx -y @wendellresearch-ai/benchwarmers mcp without installing, or python mcp_server.py from a
checkout, a small process per client that calls the same server. For example:
{"mcpServers": {"benchwarmers": {"command": "npx", "args": ["-y", "@wendellresearch-ai/benchwarmers", "mcp"],
"env": {"BENCHWARMERS_URL": "http://127.0.0.1:8122"}}}}On native Windows, a client that cannot start npx itself needs
"command": "cmd", "args": ["/c", "npx", "-y", "@wendellresearch-ai/benchwarmers", "mcp"].
Variable | Meaning |
| the server, default |
| a gateway key from Settings, sent as |
| seconds to wait for a panel, default 900 |
Behind an MCP aggregator (1mcp, MetaMCP, mcp-proxy, Docker's MCP Gateway
and the like): add http://127.0.0.1:8122/mcp as an HTTP (streamable HTTP)
upstream, with the Authorization header where keys are required; an
aggregator that only runs stdio servers takes benchwarmers mcp as above.
Give it a request timeout of a few minutes or more, as a panel takes that
long; 1mcp's default of 60 seconds ends most of them. With 1mcp, for example:
1mcp mcp add benchwarmers --type http --url http://127.0.0.1:8122/mcp --timeout 900000The aggregator prefixes the tool names (1mcp's are
benchwarmers_1mcp_ask and so on). If you also use the Claude Code plugin,
disable the plugin's own benchwarmers server in /mcp so the tools are not
listed twice; the skill and /benchwarmers:ask keep working, though the
command then asks before each tool call, since its pre-approved tools carry
the plugin's names. The plugin's verdict hook calls the plugin's own server,
so it does not work through an aggregator: with that server disabled it only
shows a hook-error notice at the end of each turn.
An agent running on a collection. The MCP tools ask a panel for a
second opinion; an agent can also be driven by a collection, in route
mode (bench/<collection>:route): one member, picked by its record, gets
each turn with its tools, stays on for the whole conversation, and is
replaced by the next member if it fails before answering. Claude Code
points at it with ANTHROPIC_BASE_URL=http://127.0.0.1:8122 and
ANTHROPIC_MODEL=bench/core:route; Codex, opencode, Cline and other
OpenAI-, Responses- or Ollama-speaking agents with the same base URL.
GATEWAY.md, "Route mode", has the set-up and the limits.
Ready set-ups. The dashboard's Connect page (Gateway) gives each popular app's configuration, filled in from your install's address, collections and MCP server, with a Copy button: opencode, VS Code chat, Codex CLI, Zed, Continue, Cline and Kilo Code, Claude Code, your own Python with LangChain or the OpenAI Agents SDK, goose, and chat apps (LobeHub, AnythingLLM, Cherry Studio, Chatbox, LibreChat). You choose how the app uses Benchwarmers (a collection as its model, the panel as tools, or both), which collection, and its name; Set it up writes the files for opencode, VS Code, Zed, Codex, Continue and goose itself, after showing them, with a copy of each file it changes; and Does it work? shows the app's first call. An app may have its own key, made there for its collection; it goes into the lines you copy, never into a file that may sync. The same set-ups are in CLIENTS.md, with a fresh install's address and collections, for reading here or handing to an agent.
The suite
Models are scored on review work: find the faulty line, or say whether a claim about some code is true. The repository ships a starter pack:
Real bugs (
suite/tasks/rebench-*): recent fixes from public projects, drawn from the SWE-rebench-V2 dataset, each paired with a control.Claims (
suite/tasks/claim-*): true and false statements about this project's own code, with the code attached.
Add your own from your repositories (config/repos.json in the data folder
maps a name to a folder): python suite/seed.py --repo <name> plants
realistic bugs in your real functions, each with a clean control. Private
code makes the best tasks, because no model has trained on it. See
suite/README.md, suite/REBENCH.md and
BENCHMARKS.md.
Docker and hosting
One container runs the dashboard, the gateway (on the same port, at /v1)
and the scheduler, with the data folder as a volume and no llm CLI (the
built-in client makes every call). Put the secrets in a .env file beside
compose.yaml:
BENCHWARMERS_ADMIN_TOKEN=<16 or more characters>
BENCHWARMERS_KEY_OPENROUTER=sk-or-...then docker compose up -d and open http://127.0.0.1:8122. The compose file
publishes on this machine only; see its comments for a proxy.
Hosted, always on: HOSTING.md compares Fly.io, Railway, Render and a VPS with Caddy (cost, sleep, disks, HTTPS), with a template for each under deploy/.
Render is one click: a paid instance with a 1 GB disk and a generated admin
token. Fly.io is four commands from a checkout (cp deploy/fly.toml .,
fly launch --copy-config --no-deploy, fly secrets set BENCHWARMERS_ADMIN_TOKEN=..., fly deploy). Railway is a few steps in its CLI
or browser, in HOSTING.md.
What leaves your machine
Benchwarmers has no account, no telemetry and no server of its own; nothing reaches Wendell Research. What it sends, and where:
Your prompts, only to the endpoints you add: a provider, an aggregator, a hub such as Open WebUI, or a local runtime. That includes the probes, the suite and onboarding, which spend calls on your keys, and the decision models' daily benchmark, which puts past requests, judged answers and redacted error text to them again. Add only endpoints you are content to send that work to.
Catalog reads, no key and nothing of yours: FreeLLMAPI (its feed and its site's sitemap) and models.dev once a day, and OpenRouter's public model list. With an NVIDIA connection, the public file of rate limits build.nvidia.com states, once a day and on Refresh.
Model lists, with no prompt: each Open WebUI connection's list of the models its provider serves, asked through Open WebUI with the key it already holds, once a day and on Discover's Refresh lists.
Sign-ins, only when you choose one: the provider's own sign-in page.
Quota reads, only when you press Refresh on Quota management, with no prompt: OpenRouter's key status, and xKiro's and Requesty's usage, each with your key for it (from its Open WebUI connection when no endpoint has one). CLI Proxy API or OmniRoute asks each sign-in's own provider for its quota. freellmapi, one-api and new-api are asked themselves, on your machine or LAN; one-api and new-api also when the Onboard dialog opens, for their channels.
Charts: the dashboard's browser page loads Observable Plot from jsDelivr at a pinned version.
Posts, only when you click Post on X, Bluesky, LinkedIn or Reddit in a report's Share menu: that site opens in your browser with the report's name and the project's link. The image goes only to your clipboard, a file, or the app you pick in your system's share sheet.
Seeding real-bug tasks, only when you run
suite/rebench.py: the SWE-rebench-V2 rows from Hugging Face and the files from GitHub.
Keys stay in the environment or llm's key file (keys.json in its user
folder), not in the data folder. An export (Settings, Export and import)
carries them only if you tick them, and an import that carries keys first
backs up the ones it replaces, in the data folder's backups/.
PRIVACY.md and SECURITY.md have the detail.
Limits
Tools go through route mode only. A panel answers with text; a request with tools is refused there, not trimmed. Route mode passes turns only to members on chat-completions endpoints, keeps its sticky conversations in memory (a restart starts each on a fresh pick), and has been run end to end with Claude Code and opencode 2; the other set-ups on the Connect page are written from each app's docs, not run (Codex's profile among them). Shadow seats (another member answering an agent's turn afterwards, for evaluation) are opt-in (
shadow_rateinconfig/gateway.json) and compare the shape of the answers (text or tool calls, which tools, arguments that parse), not whether either was right.Some parts are checked against fakes only. The native Anthropic and Gemini endpoints, and the LiteLLM, CLI Proxy API, OmniRoute, freellmapi, one-api and new-api hub adapters, are built from those APIs' documentation and source and tested against fake servers, not yet against the real services.
A merged reply arrives in one piece once the merger has finished; the seats before it stream as they finish.
A late seat's answer is not in the reply. A seat still answering a minute after two others did is named
late; its answer reaches the dashboard and its call idpending_verdicts, not the caller.quorumandgraceinconfig/gateway.jsonchange that (GATEWAY.md, "The reply").Pacing and the ledger cover the built-in client. Calls made through the
llmCLI, when you choose it, are neither paced nor counted.Calls cost. Probes, onboarding, the suite and the decision benchmark run on your keys and free tiers. Onboarding runs one model at a time for that reason (Quick spreads the suite over the scheduled passes); a suite pass is paced, not burst.
The scheduler lives in the server. Its jobs run while the dashboard (or the container, or the service) is up; a missed run is made up once, within seven days, when it comes back.
One install, one person at heart. Logins and gateway keys exist for a LAN or a server, but there are no per-user views or roles.
The onboarding queue lives in the dashboard's memory. Restarting it drops the rest of a batch; each started model's log stays in
runs/.Free tiers change without notice. Discover's prices come from public catalogs, and some providers' free models work only in their own app (switched per provider on Settings).
Security
Both servers listen on 127.0.0.1 by default and accept only same-origin writes
for their own host name, so a web page in your browser cannot drive them. To
listen anywhere else, set BENCHWARMERS_ADMIN_TOKEN first; without it they
refuse to start. With it, the dashboard asks for a login and the gateway takes
named, revocable gateway keys. Settings, under Server and access, shows how
this stands (whether a secret is set, never its value).
SECURITY.md has the whole model and how to report a vulnerability.
Development
python scripts/check.py # every self-check and test (--quick skips the slow one)The core is the Python standard library only. CI runs the checks on Linux and Windows for Python 3.11 to 3.13, and on macOS.
Benchwarmers isn't taking outside contributions yet; that opens once it is more established. A security problem goes through SECURITY.md.
Licence
MIT, Copyright (c) 2026 Wendell Research (LICENSE). The starter tasks and bundled fonts carry their own licences, listed in NOTICE. Use is also covered by TERMS.md.
Available Tools
4 toolsaskAsk a collectionA
Put a prompt to every seat of one collection at once and get every answer back, each under its seat and model, plus the seats as data (alias, seat, status, call id). Can take minutes. Ask a collection, not a favourite model. Give the seats the real text (code, diff, file contents) in the prompt: they cannot read your files. Verify every claim against the source before acting on it, name every seat that answered in what you tell the user, then send a verdict per seat you judged.
Inputs: prompt (required), collection (required), with, merged, +1 more.
list_collectionsList collectionsARead-only
The collections this Benchwarmers server offers: name, what each is for, tags, members, explorer seats, reply mode, and what the members handle together (smallest context window, shared capabilities, tightest rate limits). Pick by task before calling ask. A collection the user names may not be listed here: ask it anyway.
Inputs: no named parameters.
pending_verdictsSeats still to judgeARead-only
For the Benchwarmers Claude Code plugin's Stop hook, which calls it at the end of a turn: the seats of this session's panels that have no verdict yet. You do not need to call it; send verdicts with verdict.
Inputs: active, session, transcript.
verdictRecord a verdictAIdempotent
Judge one seat's answer by its call id, once you have checked it against the source. One per seat you judged. List the claims it got right in real and the claims that are untrue in false, one sentence each: they feed routing and the 'confidently wrong' record, so a note alone is not enough.
Inputs: call (required), useful (required), note, real, +1 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
TDQS
Scored across 4 tools
Each tool targets a distinct action: enumerate collections, query a collection, list seats awaiting judgment, and submit a judgment. The pending_verdicts/verdict pair is complementary (read vs. write) and clearly separated, though the note that pending_verdicts is for a Stop hook and 'you do not need to call it' adds mild confusion about whether an agent should invoke it at all.
All names use snake_case, so the casing convention is uniform, but the verb style is mixed: `list_collections` is verb_noun, `ask` is a bare verb, while `verdict` and `pending_verdicts` are nouns with no verb at all. Readable, but a reader cannot predict a tool's shape from its name.
Four tools match the narrow domain of querying model collections and recording judgments; each has a clear job (discover, ask, check pending, judge). It is on the lean side and leans on server-side configuration for collections, but no tool feels redundant or missing at this scope.
The core loop (list collection → ask → verdict) is covered, and call ids are surfaced in ask output. However, there is no tool to retrieve a prior call's answers by call id, no way to inspect a collection's members in detail, and no way to review already-submitted verdicts or the 'confidently wrong' record — so the call-id datum is a dead end for the agent.
Maintenance
Related MCP Connectors
OpenAI-compatible BYOK LLM gateway: per-task routing, shadow evaluation, promotion gates.
AI model routing on your own vendor keys: pick the best model per prompt, or route and run it.
Open-weight and abliterated LLMs on one API: compare models and rankings, chat, check credits.
Self-hosted AI prompt library: prompts, collections, tags, teams, chains. 29 MCP tools for agents.
Related MCP Servers
- AlicenseNot gradedqualityDmaintenanceAn MCP server that queries a panel of LLMs from different providers via OpenRouter and returns their answers side by side, optionally synthesizing them to highlight disagreements.1MIT
- AlicenseNot gradedqualityCmaintenanceA self-hostable MCP server that routes prompts to multiple LLM providers using declarative policies, with multi-role orchestration for independence and verification.MIT
- AlicenseAqualityBmaintenanceBlind multi-model councils with anonymous LLM seats and direct 1:1 chat as a local MCP server. Enables asking one question to get independent answers from multiple model CLIs, then blind scoring and reveal.1759 PyPIAGPL 3.0
- AlicenseNot gradedqualityBmaintenanceSelf-hosted MCP gateway that aggregates many MCP servers into a single endpoint for AI clients, supporting hosted, local stdio, and Git-sourced MCPs.3MIT