Benchwarmers
Server Configuration
Describes the environment variables required to run the server.
| Name | Required | Description | Default |
|---|---|---|---|
| BENCHWARMERS_KEY | No | A gateway key from Settings, sent as Authorization: Bearer; needed for a server on another machine, or where keys are required. | |
| BENCHWARMERS_URL | No | The Benchwarmers server URL. A trailing /v1 is fine. | http://127.0.0.1:8122 |
| BENCHWARMERS_TIMEOUT | No | Seconds to wait for a panel. | 900 |
Instructions
Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.
This server publishes no instructions, or was last inspected before Glama recorded them.
Capabilities
Features and capabilities supported by this server
Protocol revision2025-11-25
| Capability | Details |
|---|---|
| tools | {
"listChanged": false
} |
Tools
Functions exposed to the LLM to take actions
| Name | Description |
|---|---|
| list_collectionsA | The collections this Benchwarmers server offers: name, what each is for, tags, members, explorer seats, reply mode, and what the members handle together (smallest context window, shared capabilities, tightest rate limits). Pick by task before calling ask. |
| askA | Put a prompt to every seat of one collection at once and get every answer back, each under its seat and model, plus the seats as data (alias, seat, status, call id). Can take minutes. Ask a collection, not a favourite model. Give the seats the real text (code, diff, file contents) in the prompt: they cannot read your files. Verify every claim against the source before acting on it, name every seat that answered in what you tell the user, then send a verdict per seat you judged. |
| verdictA | Judge one seat's answer by its call id, once you have checked it against the source. One per seat you judged. List the claims it got right in |
| pending_verdictsA | For the Benchwarmers Claude Code plugin's Stop hook, which calls it at the end of a turn: the seats of this session's panels that have no verdict yet. You do not need to call it; send verdicts with |
Prompts
Interactive templates invoked by user choice
| Name | Description |
|---|---|
No prompts | |
Resources
Contextual data attached and managed by the client
| Name | Description |
|---|---|
No resources | |
TDQS
Scored across 4 tools
Each tool targets a distinct step: discover collections, query a panel, submit a judgement, and enumerate unjudged seats. The only soft boundary is between `verdict` and `pending_verdicts`, which the description partially resolves by telling the agent it never needs to call the latter, though that still invites hesitation.
`list_collections` uses a verb_noun pattern while `ask` and `verdict` are bare action words and `pending_verdicts` is a noun phrase. The names are readable and memorable, but there is no single predictable convention an agent can rely on.
Four tools is lean but matches the narrow discover→ask→judge loop, with no redundant entries. `pending_verdicts` exists mainly for a plugin hook rather than agent use, so the effective agent-facing surface is even smaller, which is acceptable but slightly thin.
The core lifecycle is covered: enumerate collections, fan a prompt to all seats, and record per-seat correctness feedback that feeds routing. Gaps are minor: no way to inspect prior calls or verdicts, cancel a long-running `ask`, or query the 'confidently wrong' record that verdicts supposedly populate.