Skip to main content
Glama

Server Configuration

Describes the environment variables required to run the server.

NameRequiredDescriptionDefault
BENCHWARMERS_KEYNoA gateway key from Settings, sent as Authorization: Bearer; needed for a server on another machine, or where keys are required.
BENCHWARMERS_URLNoThe Benchwarmers server URL. A trailing /v1 is fine.http://127.0.0.1:8122
BENCHWARMERS_TIMEOUTNoSeconds to wait for a panel.900

Instructions

Guidance the server publishes about itself, which clients place ahead of the tool catalog so the model reads it before choosing anything.

Benchwarmers asks a COLLECTION of models at once (every member plus explorer seats it rotates in) and keeps every answer. Rules: (1) ask a collection, not a favourite model; list_collections says what each is for. (2) Verify before acting on anything a seat says: seats fail together, so agreement is not evidence. (3) Send a verdict per seat you judged, with the claims it got right in real and the untrue ones in false. (4) Name every seat that answered, by model and seat, including errors, empty replies and late seats. An empty reply from a reasoning model is usually its token budget, not incapacity. Good uses: exhaustive enumeration over real files, adversarial review. Keep the design and the decisions yourself.

Capabilities

Features and capabilities supported by this server

Protocol revision2025-11-25

CapabilityDetails
tools
{
  "listChanged": false
}

Tools

Functions exposed to the LLM to take actions

NameDescription
list_collectionsA

The collections this Benchwarmers server offers: name, what each is for, tags, members, explorer seats, reply mode, and what the members handle together (smallest context window, shared capabilities, tightest rate limits). Pick by task before calling ask. A collection the user names may not be listed here: ask it anyway.

askA

Put a prompt to every seat of one collection at once and get every answer back, each under its seat and model, plus the seats as data (alias, seat, status, call id). Can take minutes. Ask a collection, not a favourite model. Give the seats the real text (code, diff, file contents) in the prompt: they cannot read your files. Verify every claim against the source before acting on it, name every seat that answered in what you tell the user, then send a verdict per seat you judged.

verdictA

Judge one seat's answer by its call id, once you have checked it against the source. One per seat you judged. List the claims it got right in real and the claims that are untrue in false, one sentence each: they feed routing and the 'confidently wrong' record, so a note alone is not enough.

pending_verdictsA

For the Benchwarmers Claude Code plugin's Stop hook, which calls it at the end of a turn: the seats of this session's panels that have no verdict yet. You do not need to call it; send verdicts with verdict.

Prompts

Interactive templates invoked by user choice

NameDescription

No prompts

Resources

Contextual data attached and managed by the client

NameDescription

No resources

TDQS

A4/5.0

Scored across 4 tools

Disambiguation4/5

Each tool targets a distinct action: enumerate collections, query a collection, list seats awaiting judgment, and submit a judgment. The pending_verdicts/verdict pair is complementary (read vs. write) and clearly separated, though the note that pending_verdicts is for a Stop hook and 'you do not need to call it' adds mild confusion about whether an agent should invoke it at all.

Naming Consistency3/5

All names use snake_case, so the casing convention is uniform, but the verb style is mixed: `list_collections` is verb_noun, `ask` is a bare verb, while `verdict` and `pending_verdicts` are nouns with no verb at all. Readable, but a reader cannot predict a tool's shape from its name.

Tool Count4/5

Four tools match the narrow domain of querying model collections and recording judgments; each has a clear job (discover, ask, check pending, judge). It is on the lean side and leans on server-side configuration for collections, but no tool feels redundant or missing at this scope.

Completeness3/5

The core loop (list collection → ask → verdict) is covered, and call ids are surfaced in ask output. However, there is no tool to retrieve a prior call's answers by call id, no way to inspect a collection's members in detail, and no way to review already-submitted verdicts or the 'confidently wrong' record — so the call-id datum is a dead end for the agent.

Maintenance

ActivityMaintained
ResponsivenessNo recent community issues