Benchwarmers
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| BENCHWARMERS_KEY | No | A gateway key from Settings, sent as Authorization: Bearer; needed for a server on another machine, or where keys are required. | |
| BENCHWARMERS_URL | No | The Benchwarmers server URL. A trailing /v1 is fine. | http://127.0.0.1:8122 |
| BENCHWARMERS_TIMEOUT | No | Seconds to wait for a panel. | 900 |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
Benchwarmers asks a COLLECTION of models at once (every member plus explorer seats it rotates in) and keeps every answer. Rules: (1) ask a collection, not a favourite model; list_collections says what each is for. (2) Verify before acting on anything a seat says: seats fail together, so agreement is not evidence. (3) Send a verdict per seat you judged, with the claims it got right in real and the untrue ones in false. (4) Name every seat that answered, by model and seat, including errors, empty replies and late seats. An empty reply from a reasoning model is usually its token budget, not incapacity. Good uses: exhaustive enumeration over real files, adversarial review. Keep the design and the decisions yourself.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| list_collectionsA | The collections this Benchwarmers server offers: name, what each is for, tags, members, explorer seats, reply mode, and what the members handle together (smallest context window, shared capabilities, tightest rate limits). Pick by task before calling ask. A collection the user names may not be listed here: ask it anyway. |
| askA | Put a prompt to every seat of one collection at once and get every answer back, each under its seat and model, plus the seats as data (alias, seat, status, call id). Can take minutes. Ask a collection, not a favourite model. Give the seats the real text (code, diff, file contents) in the prompt: they cannot read your files. Verify every claim against the source before acting on it, name every seat that answered in what you tell the user, then send a verdict per seat you judged. |
| verdictA | Judge one seat's answer by its call id, once you have checked it against the source. One per seat you judged. List the claims it got right in |
| pending_verdictsA | For the Benchwarmers Claude Code plugin's Stop hook, which calls it at the end of a turn: the seats of this session's panels that have no verdict yet. You do not need to call it; send verdicts with |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 4 tools
Each tool targets a distinct action: enumerate collections, query a collection, list seats awaiting judgment, and submit a judgment. The pending_verdicts/verdict pair is complementary (read vs. write) and clearly separated, though the note that pending_verdicts is for a Stop hook and 'you do not need to call it' adds mild confusion about whether an agent should invoke it at all.
All names use snake_case, so the casing convention is uniform, but the verb style is mixed: `list_collections` is verb_noun, `ask` is a bare verb, while `verdict` and `pending_verdicts` are nouns with no verb at all. Readable, but a reader cannot predict a tool's shape from its name.
Four tools match the narrow domain of querying model collections and recording judgments; each has a clear job (discover, ask, check pending, judge). It is on the lean side and leans on server-side configuration for collections, but no tool feels redundant or missing at this scope.
The core loop (list collection → ask → verdict) is covered, and call ids are surfaced in ask output. However, there is no tool to retrieve a prior call's answers by call id, no way to inspect a collection's members in detail, and no way to review already-submitted verdicts or the 'confidently wrong' record — so the call-id datum is a dead end for the agent.