What can you do with this server?

An MCP server (over stdio) for building and running custom LLM tournaments: define benches, preview costs, run multi-judge evaluations, and read auditable results. * Discover options free: tournament_options lists benches, scenarios, model tiers with prices, judge personas, provider status, defaults, and limits; tournament_list_benches reads local bench/scenario IDs. * Preview before spending: tournament_plan_run resolves your choices against the live catalog, fills defaults, and returns the plan, rough USD estimate, warnings, and readyToRun — no model calls, no run folder. * Run a cheap check: tournament_quick_test does one model, one scenario, one judge, with optional judge model/persona and 1–10 turns (paid). * Run a full tournament: tournament_evaluate compares 1–4 models across chosen scenarios with 1–5 judge seats (named personas or custom lenses), optional synthesizer and simulated participant; streams progress and takes minutes (paid). * Read saved results locally: tournament_leaderboard (best score per model, filterable by bench) and tournament_get_run (full run: judges, per-scenario scores, confidence/outliers, failures). * Create reusable benches: tournament_create_bench saves scenarios, prompts, criteria, rounds, and optional participant persona as a new bench usable immediately as a plugin. * Use all three MCP primitives: tools plus resources (tournament://benches, leaderboard, runs, runs/{runId}, runs/{runId}/report) and prompts (setup_tournament, compare_models, choose_model_for_task, explain_run).

How do I use mcp-tournament?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@mcp-tournament Run the business-strategy benchmark on my model" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

mcp-tournament

by samalbanese

Overview Schema Source Related Servers Discussions Admin

TypeScript

Remote

mcp-tournament

License: MIT TypeScript Node >= 20 MCP server

https://github.com/user-attachments/assets/12b0a01c-1592-465f-927a-ba421ec32ab7

In this study, all five AI judges went easier on their own lab's answers than on everyone else's. Five frontier models answered twelve business scenarios and then judged each other blind. Read the findings.

Build a custom LLM benchmark in a form, run it from a local GUI, MCP client, or CLI, and turn independent judge opinions into ranked, auditable results.

Study findings: which AI is best at real business work, and can you trust AI judges to tell you?

Explore the interactive demo: compare a recorded business-strategy experiment, inspect the evidence, and try your own criterion weights. No key needed.

Why this is interesting:

Disagreement is data: multiple specialist judges score independently; the arbiter preserves outliers and explains where they diverged.
Benches are declarative: anyone can define scenarios and criteria as JSON or build them in a form, no pipeline code required.
BYOK and local-first: bring one OpenRouter key, keep the GUI on your machine, and run budget-tier tournaments for cents.

The flagship study

Claude Opus 5.5, GPT-6.1 Sol, Gemini 3.1 Pro, DeepSeek V4 Pro and Qwen3.8 Max each answered the same twelve scenarios across business strategy, customer support and creative writing. Every model then scored every answer, its own included, without seeing who wrote it: 300 scorecards in all.

Every judge favored its own lab. Compared with the rest of the panel, each judge scored its own lab's answers more generously than it scored the other four labs. The lift ranged from about 0.3 points (DeepSeek) to 1.4 points (GPT) on a 10-point scale, and every 95% interval sits above zero.
No clear winner at the top. GPT-6.1 Sol led at 8.4, but its interval overlaps those of Claude Opus 5.5 (8.2) and DeepSeek V4 Pro (8.0), so the study can't separate the three.
One judge alone picks a different winner. GPT and Claude each named themselves best when judging alone, and the panel agreed only weakly overall (Krippendorff's alpha 0.30).

Self-preference heatmap: each judge's score offset from the rest of the panel, by candidate family

The findings page links every number back to the transcripts and judge reasoning behind it, and the raw scores download as CSV. Recorded API cost: $7.25. Results apply to these scenarios and settings; the page lists what the study does not show. To run your own, see the study format.

Related MCP server: metrillm-mcp

A model choice you can explain

The evaluation studio turns the recorded pipeline output into a decision workflow:

Results overview: the original leaderboard, criterion comparisons, and material judge dissent in one place.
Compare evidence: choose up to three candidates and inspect their original answers and arbiter assessments side by side.
Decision lab: change the importance of each criterion with sliders or presets, then see the weighted ranking respond immediately.
Export a report: download original scores, judge identities, dissent, and optional exploratory weights as Markdown.
Follow the evidence: open any model's scorecard, individual judge matrix, transcript, or animated run replay.

Decision lab with adjustable priorities and explicitly separated recorded scores

The demo contains real, dated recordings across business strategy, customer support, creative writing, and the D&D showcase. It makes no live model calls. Results are scenario-specific observations, not a statistically validated or universal model ranking. The decision lab recomputes scores from final criteria; it never changes recorded results or hides missing evidence. Candidate and judge models may overlap.

How it works

flowchart LR
    A["Scenario + criteria<br/>plugin / bench JSON"] --> B["EXECUTE<br/>candidate + tool calls"]
    B --> C["JUDGE<br/>N specialists in parallel"]
    C --> D["SYNTHESIZE<br/>merge + flag outliers<br/>never scores independently"]
    D --> E["AGGREGATE<br/>leaderboard + JSON audit trail"]

Three entry points feed the same pipeline:

GUI: build benches, launch runs, and inspect results locally.
MCP client: ask Claude Desktop, Cursor, or Windsurf to run, compare, and explain evaluations (8 tools, 5 resources, 4 prompts), including saving your own benches from chat.
CLI: script runs, serve MCP over stdio, or print the leaderboard.

Domain logic is pluggable; the pipeline is not. Benches are declarative plugins: a JSON file (or the Build Bench form) defines scenarios, rounds, an optional simulated participant persona, and judging criteria. Code plugins can go further with custom tools; see docs/PLUGINS.md.

Plugin	Domain	Kind
`business-strategy`	SMB pricing decision with real numbers to reason about	📄 bench (JSON)
`creative-writing`	Opening chapter + 3 rounds with a developmental-editor persona	📄 bench (JSON)
`customer-support`	Billing dispute with an escalating customer persona	📄 bench (JSON)
`dnd`	Showcase: D&D 5e Dungeon Master with dice/damage tools and an LLM player	⚙️ code plugin
`coding`	Code generation & review	⚙️ code plugin
Yours	Build in the GUI (`#/build`), drop a JSON in `benches/`, or write TypeScript	🛠 you

Why multi-judge?

Single evaluators miss things. A Rules judge catches mechanical errors; a Creative judge catches boring output; a Holistic judge catches "would I keep using this?" The synthesizer never scores independently: it arbitrates, flags outlier judges, and records why they disagreed. Judge disagreements are first-class data, rendered in the viewer:

Model scorecard with judge disagreements

When to use this (and when not to)

You want	Reach for
CI-style assertions and regression gates over prompts at scale	promptfoo
Standardized academic benchmarks (MMLU, HellaSwag, …)	lm-eval-harness
Rubric-scored comparisons on your own scenarios (multi-round conversations, personas, tool use) with judge disagreement preserved instead of averaged away	mcp-tournament

Those tools are better at what they do; this one is for judgment-heavy, domain-specific evals where a single aggregate score hides the story.

Quick start

git clone https://github.com/relaywright/mcp-tournament.git
cd mcp-tournament
npm run setup                          # installs + builds server and GUI
export OPENROUTER_API_KEY=sk-or-...    # one key, every role

Or skip local setup entirely: Open in GitHub Codespaces

As a local app (BYOK GUI)

node dist/cli.js gui              # http://localhost:4600

Paste your OpenRouter key in Settings (stored in your browser, sent only to this local server, never written to disk), then set your model routing right below it (default candidates from the live catalog with prices, plus the model behind each judge and the synthesizer) and start a run from NEW RUN. BUILD BENCH creates a new benchmark from a form (question, rounds, persona, judging criteria, with an AI-suggest button) and saves it as a JSON plugin, live immediately.

As a desktop app (Windows, unsigned preview)

The same server + GUI wrapped in an Electron window, with the API key stored via OS-level encryption (safeStorage) instead of the browser:

npm --prefix electron install
npm --prefix electron run dist   # unsigned NSIS installer + portable exe → electron/dist-app/

Builds are unsigned for now, so Windows SmartScreen will warn on first run; see electron/README.md.

As an MCP server (Claude Desktop, Cursor, Windsurf)

{
  "mcpServers": {
    "tournament": {
      "command": "node",
      "args": ["<path-to-repo>/dist/index.js"],
      "env": {
        "OPENROUTER_API_KEY": "sk-or-...",
        "TOURNAMENT_RESULTS_DIR": "<path-to-repo>/results"
      }
    }
  }
}

MCP clients start the server from their own working directory, so TOURNAMENT_RESULTS_DIR is what lets it find (and add to) the repo's saved runs.

The server runs over stdio and uses all three MCP primitives: tools, resources, and prompts.

Tool	What it does	Cost
`tournament_options`	Benches, model shortlist with prices, personas, provider status, defaults, and limits	Free catalog lookup, no model calls
`tournament_plan_run`	Resolved settings, summary, rough cost, and setup warnings	Free catalog lookup, no model calls
`tournament_list_benches`	Every bench and its scenario IDs	Free, local read
`tournament_leaderboard`	Best score per model across saved runs, optionally per bench	Free, local read
`tournament_get_run`	One saved run in full: models, judges, per-scenario scores, failures	Free, local read
`tournament_quick_test`	One model, one scenario, one judge; optional judge persona and turns	Paid through the selected provider
`tournament_evaluate`	1–4 models across selected scenarios; choose judge models, personas, turns, and simulated user	Paid through the selected providers, several minutes
`tournament_create_bench`	Saves a new bench (your scenarios and scoring criteria), usable immediately	Free, local write

Every tool returns a readable markdown answer plus typed structuredContent that matches a declared outputSchema. Tools carry annotations (readOnlyHint, openWorldHint) so a client can tell a free read from a paid run, and the two paid tools stream notifications/progress (one step as each model/scenario pair starts and finishes) so long runs don't look frozen. The server also sends connection-time instructions telling the assistant to confirm with you before spending money.

Your own scenarios and judges. Describe what you want tested and the assistant drafts a bench (scenarios, a prompt for each, and the criteria judges score against), shows it to you, then saves it with tournament_create_bench. The bench is written to benches/ and works right away as plugin: "<name>". tournament_evaluate accepts judgeModels (one model ID per judge seat, in the order Rules, Creative, Holistic, Authentic Voice, Context; the list length sets the panel size) and synthesizerModel for the model that reconciles their scores.

Resources (data a client can attach without a tool call):

URI	Contents
`tournament://benches`	Benches and scenarios (JSON)
`tournament://leaderboard`	All-time best score per model (JSON)
`tournament://runs`	Index of saved runs with each winner (JSON)
`tournament://runs/{runId}`	One run in full (JSON)
`tournament://runs/{runId}/report`	One run as a markdown scorecard

The two templates list every saved run and autocomplete run IDs.

Prompts (slash commands in clients that support them):

setup_tournament (optional goal): shows options, asks only for missing choices, previews cost, and gets your yes before a run.
compare_models (optional models, plugin, judges): compares selected models and explains who won; starts guided setup when models are missing.
choose_model_for_task (task): matches the task to a bench and checks existing results first, asking before any paid run.
explain_run (runId): attaches a run's scorecard and asks for a plain-English explanation.

Once it's connected, you can just ask:

"Which benches does mcp-tournament have, and who leads the customer-support leaderboard?"
"Quick-test deepseek/deepseek-v3.2 on the coding bench."
"Compare deepseek/deepseek-v3.2 and openai/gpt-5.4-mini on business-strategy."
"Make a bench that tests how models handle a customer disputing a late fee, then run deepseek/deepseek-v3.2 against openai/gpt-5.4-mini with qwen/qwen3.5-flash-02-23 and deepseek/deepseek-v3.2 as the judges."

As a CLI

# The demo: 3 cheap models, 1 bench scenario, 3 judges (~a few cents)
node dist/cli.js run --plugin business-strategy \
  --models "deepseek/deepseek-v3.2,google/gemini-2.5-flash-lite,meta-llama/llama-4-scout" \
  --scenario pricing-pivot --judges 3

# Or the tool-calling showcase: D&D DM with dice/damage tools and an LLM player
node dist/cli.js run --plugin dnd --models "deepseek/deepseek-v3.2" \
  --scenario dnd-combat --judges 3

node dist/cli.js leaderboard
node dist/cli.js serve          # MCP stdio server

Swapping models, judges, scenarios, and turns

In your MCP client, start with the setup_tournament prompt or ask to set up a tournament for your goal. The assistant calls tournament_options to show benches and scenario counts, a model shortlist grouped by price tier, judge personas, and which provider accounts are ready. Any OpenRouter model ID works, including models outside the shortlist. Prices are USD per million input or output tokens.

The assistant asks only for missing choices, one short question at a time: the bench and 1-4 candidate models. It can help create a bench with tournament_create_bench. The remaining defaults are fine unless you want different judges, personas, turns, a synthesizer, or a simulated user. Settings apply to this run.

For example, call tournament_plan_run with:

{
  "bench": "customer-support",
  "scenarios": ["billing-dispute"],
  "candidates": ["deepseek/deepseek-v3.2", "openai/gpt-5.4-mini"],
  "judgePanel": [
    { "persona": "skeptic", "model": "qwen/qwen3.5-flash-02-23" },
    {
      "model": "google/gemini-2.5-flash-lite",
      "customPersona": {
        "name": "Support customer",
        "lens": "Check whether the customer gets a clear next step without repeating information."
      }
    }
  ],
  "turns": 2,
  "synthesizer": "deepseek/deepseek-v3.2",
  "participant": "deepseek/deepseek-v3.2"
}

The preview fills in defaults, checks model IDs against the live catalog, and shows a rough estimate such as ≈ $0.09 (rough, could be ±50%). Models without prices are listed as excluded. If the catalog is offline, discovery uses a curated fallback and the cost estimate is unavailable. Previewing makes no model calls and creates no run folder. Missing provider keys appear as setup warnings.

After you say yes, the assistant calls tournament_evaluate with those same choices. The preview's bench, candidates, synthesizer, and participant fields become plugin, models, synthesizerModel, and participantModel in the evaluate tool. scenarios, judgePanel, and turns keep their names. Clients that support forms also show a confirmation form. Only accepting with the confirmation box checked starts the run. Declining, cancelling, leaving it unchecked, or a form error returns: "Cancelled. Nothing was run and nothing was charged."

Clients without form support run immediately when tournament_evaluate is called, so the assistant must get your yes in chat first. tournament_quick_test is the cheap check: one model, one scenario, one judge, with optional judge and turns. It makes paid calls without a form.

Choose 1-5 judge seats. Each can set its own model and one of these personas:

Persona ID	Name	What it checks
`rules`	Accuracy	Facts, reasoning, rules, and tool use
`creative`	Craft & Clarity	Clear, original, useful communication
`holistic`	Holistic	Task completion and the overall experience
`authentic_voice`	Authentic Voice	Natural, specific language without repetition
`npc_world`	Context & Consistency	Coherent details throughout the conversation
`strict`	Strict Grader	Flaws, with high scores reserved for excellent work
`skeptic`	Skeptic	Unsupported claims and steps that would not work
`audience`	Target Audience	Whether the reader can understand, trust, and use the answer

Use either persona or customPersona on a seat. A custom lens must be 1-1000 characters, with an optional name of 1-60 characters. The default panel is Accuracy, Craft & Clarity, and Holistic. A single judge needs no synthesizer. Existing judges and judgeModels inputs still work; judgePanel takes priority if both are supplied.

Set turns to 1-10 to use the same turn count for every selected scenario. Omit it to use each scenario's default, shown by tournament_options. Omit scenarios to run all scenarios in the bench. The participant model plays the simulated user in follow-up turns.

Reasoning levels

Add @level to any model ref to set how hard that seat thinks: none, minimal, low, medium, high, xhigh or max. It works on candidates, judges, the synthesizer and the simulated user. List the same model twice to test whether extra thinking pays for itself:

{ "candidates": ["google/gemini-3.1-flash-lite@low", "google/gemini-3.1-flash-lite@high"] }

The two levels get separate leaderboard rows, labelled · low and · high. Each level is checked against that model's entry in the OpenRouter catalog before anything is spent, so a level the model doesn't offer stops the run with the levels it does accept. A ref without a level uses the provider's default and sends no reasoning setting. Seats that think get a larger output allowance, and the cost estimate adds hidden reasoning tokens for each level. In the GUI, a Thinking menu appears only on models that offer levels.

Provider accounts

OpenRouter is the default for every role. Set OPENROUTER_API_KEY in the shell or your MCP client's server configuration. A bare model ID such as deepseek/deepseek-v3.2, or an explicit openrouter: prefix, uses that account.

Optionally set ANTHROPIC_API_KEY and use a ref such as anthropic:claude-haiku-4-5 for a candidate, judge, synthesizer, or simulated user. These Claude calls are billed per use to your Anthropic API account. This is separate from a Claude subscription. The default route stays OpenRouter, and an OpenRouter ID such as anthropic/claude-haiku-4.5 still bills OpenRouter. Only Claude model IDs work with anthropic:. Discovery reports account readiness without showing key values.

Support for running OpenAI models on a ChatGPT plan is coming soon. For now, use their OpenRouter IDs, such as openai/gpt-5.4-mini.

Results viewer

gui/ is a self-contained Vite + React static site with no backend; it deploys to any static host (Cloudflare Pages works as-is). It reads committed run JSON and renders rankings, per-judge breakdowns, disagreement callouts, and full transcripts with tool-call inspection.

Leaderboard view

cd gui && npm install
npm run import-run -- ../results/<runId>   # copy a run into the viewer
npm run build && npm run preview

Transcript view

Model routing

Every role (the candidates, each judge, the synthesizer, the participant agent) is independently model-selectable and routes through OpenRouter by default. One key, any model, no paid first-party API in the demo path. Defaults are all budget-tier (DeepSeek, Qwen Flash, Gemini Flash Lite; a full run costs cents); override per role:

TOURNAMENT_MODEL_JUDGE_RULES=openai/gpt-5.4-mini
TOURNAMENT_MODEL_SYNTHESIZER=moonshotai/kimi-k2.5
TOURNAMENT_MODEL_PARTICIPANT=deepseek/deepseek-v3.2

Values are model refs, so TOURNAMENT_MODEL_SYNTHESIZER=anthropic:claude-sonnet-5-5 sends that role to the optional Anthropic route instead of OpenRouter.

The routing layer resolves a pluggable ModelClient per role (src/clients/types.ts). OpenRouter and the optional Anthropic API route share the same pipeline. Regression tests keep all default roles on OpenRouter and the MCP server's logger on stderr (stdout is reserved for JSON-RPC).

Environment variables

Variable	Required	Purpose
`OPENROUTER_API_KEY`	For OpenRouter calls	All roles by default
`ANTHROPIC_API_KEY`	No	Optional Claude calls using `anthropic:` refs, billed per use
`TOURNAMENT_MODEL_*`	No	Per-role model overrides (see above)
`TOURNAMENT_RESULTS_DIR`	No	Results output root (default `./results`)

How it's tested

npm run test:unit runs unit tests with no API key required. The MCP layer is tested at the protocol level: a real SDK client connects over an in-memory transport and checks every tool (including saving a bench and passing chosen judges through to the pipeline), resource, template, prompt, completion, structured output, and error path, plus progress notifications from the real pipeline (strictly increasing, ending at 100%). The suite also covers the decision lab (changing priorities, zero weights, missing evidence, ties, preserved original scores, report provenance) and runs the committed demo fixtures through the viewer loaders, including known partial judge archives. Swappable-run tests cover cost estimates, catalog fallback, provider warnings, confirmation and cancellation, legacy clients, and the guided setup prompt. Two regression guards have a story:

The MCP logger writes to stderr only. stdout is reserved for JSON-RPC: one stray console.log corrupts the protocol stream and silently breaks every connected MCP client. The guard makes that a failing test instead of a mystery bug report.
The default route can never resolve to a paid first-party API. The demo path stays BYOK-through-OpenRouter at budget-tier prices. An Anthropic key is billed only when you opt in with an anthropic: ref; a config regression that would quietly bill it fails CI.

An e2e suite (npm run test:e2e) exercises real model calls when a key is present. CI runs build + unit tests + the GUI build on every push and PR.

Roadmap

Deferred deliberately: a Streamable HTTP transport for remote hosting, ChatGPT plan support, npm publish, and MCP registry submission.

Provenance

Generalized from oracle-tournament (a private project), a D&D-specific model evaluator whose pipeline proved out the multi-judge + arbiter design; this repo makes the domain pluggable.

Contributing

Issues and PRs welcome; the easiest contribution is a new bench JSON. See CONTRIBUTING.md.

License

MIT

Available Tools

8 tools

tournament_create_benchCreate a reusable evaluation benchA

Free, local, no model calls. Saves a new reusable bench (a named set of scenarios and scoring criteria) to disk. Once saved, it is immediately usable by tournament_evaluate and tournament_quick_test as plugin: "<name>", and it appears in tournament_list_benches. Bench names must be unique among registered benches. Draft the scenarios and criteria first and confirm them with the user before calling this tool, since saving registers the bench immediately.

Name	Required	Description
`name`	Yes	Unique bench name, e.g. "customer-support-escalations". 1-60 characters, must contain at least one letter or number. Fails if a bench with this name already exists.
`scenarios`	Yes	1-10 scenarios that make up this bench. Each scenario is one task the candidate model is given and scored on.
`description`	Yes	One-sentence description of what this bench tests, e.g. "Tests how well a model handles frustrated customers who feel wronged." 1-200 characters.

Output Schema

Name	Required	Description
`file`	Yes
`bench`	Yes

TDQS

A4.6/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=false, idempotentHint=false, and destructiveHint=false, so the mutation profile is covered. The description adds genuinely non-obvious context: no model calls/cost, the name must be unique among registered benches, and saving registers the bench immediately (a real side effect). It does not, however, state what happens on name collision beyond the schema's 'Fails if...', or whether a bench can be deleted/updated later.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the highest-value fact ('Free, local, no model calls'), then purpose, then integration and constraints. Three sentences mostly earn their place, though 'saving registers the bench immediately' partially repeats the opening 'Saves a new reusable bench' and the confirmation guidance.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, return values need not be explained. The description covers cost, persistence, naming constraints, downstream consumption, and the user-confirmation workflow — everything an agent needs to call a 3-param, nested-schema creation tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the baseline is 3. The description goes beyond the schema by tying the `name` parameter to its downstream usage form (plugin: "<name>") and by reinforcing the uniqueness constraint on names, which is the parameter most likely to cause a failure.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb+resource ('Saves a new reusable bench') and immediately defines what a bench is ('a named set of scenarios and scoring criteria'). It clearly distinguishes itself from sibling read tools like tournament_list_benches by naming exactly what it creates and who consumes it.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit precondition ('Draft the scenarios and criteria first and confirm them with the user before calling this tool') and names the two downstream tools (tournament_evaluate, tournament_quick_test) plus the discovery path (tournament_list_benches). An agent knows exactly when and when not to call it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

tournament_evaluateRun a full tournamentA

Costs money. Makes paid model calls and writes results to disk. Call tournament_plan_run first and get the user's yes in chat. Clients that support forms will also get a confirm form. Runs 1-4 models across a bench's scenarios and saves a leaderboard. Can take several minutes. judgePanel picks judge models and personas per seat and overrides judges and judgeModels. turns sets one turn count for every scenario. participantModel picks the simulated user model. synthesizerModel reconciles judge scores. OpenRouter is the default; anthropic: refs bill the user's Anthropic API key. Use tournament_quick_test for a cheap check.

Name	Required	Description	Default
`turns`	No	1-10 turns for every scenario. Omit to use each scenario's own default.
`judges`	No	Number of judges on the scoring panel, 1-5. Defaults to 3. Ignored when `judgeModels` is provided.
`models`	Yes	1-4 model refs to compare. Bare IDs use OpenRouter. anthropic: refs use your Anthropic API key. Add @level to set how hard a model thinks, e.g. "openai/gpt-6.1-sol@high". Levels: none, minimal, low, medium, high, xhigh, max; each model accepts only some (see tournament_options). Leave it off for the model's default. The same model may appear twice at different levels.
`plugin`	No	Bench to run, e.g. "dnd" or "coding". Defaults to "dnd".	dnd
`scenarios`	No	Scenario IDs to run. Omit to run every scenario in the bench.
`judgePanel`	No	1-5 judge seats. Each may set model and either persona or customPersona with a lens (1-1000 characters) and optional name (1-60 characters). Add @level to set how hard a model thinks, e.g. "openai/gpt-6.1-sol@high". Levels: none, minimal, low, medium, high, xhigh, max; each model accepts only some (see tournament_options). Leave it off for the model's default. The same model may appear twice at different levels.
`judgeModels`	No	1-5 OpenRouter model IDs, one per judge seat, filled in this fixed order: 1. Rules Judge (accuracy, tool_usage); 2. Creative Judge (clarity, creativity, communication); 3. Holistic Judge (overall_quality, task_completion); 4. Authentic Voice Judge (authentic_voice); 5. Context Judge (context, consistency). When provided, the panel size equals the length of this list and overrides `judges`. Omit to use the default model for each seat. Add @level to set how hard a model thinks, e.g. "openai/gpt-6.1-sol@high". Levels: none, minimal, low, medium, high, xhigh, max; each model accepts only some (see tournament_options). Leave it off for the model's default. The same model may appear twice at different levels.
`participantModel`	No	Model ref for the simulated user who sends follow-up messages. Omit for the default. Add @level to set how hard a model thinks, e.g. "openai/gpt-6.1-sol@high". Levels: none, minimal, low, medium, high, xhigh, max; each model accepts only some (see tournament_options). Leave it off for the model's default. The same model may appear twice at different levels.
`synthesizerModel`	No	OpenRouter model ID that reconciles the judges' scores into one final score. Omit to use the default synthesizer model. Add @level to set how hard a model thinks, e.g. "openai/gpt-6.1-sol@high". Levels: none, minimal, low, medium, high, xhigh, max; each model accepts only some (see tournament_options). Leave it off for the model's default. The same model may appear twice at different levels.

Output Schema

Name	Required	Description
`runId`	Yes
`judges`	Yes
`plugin`	Yes
`status`	Yes
`entries`	Yes
`message`	No
`failures`	Yes
`resultsDir`	Yes
`judgeFailures`	Yes

TDQS

A4.8/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare the generic safety profile (readOnlyHint=false, openWorldHint=true, idempotentHint=false). The description adds the details that actually change agent behavior: it costs money, makes paid model calls, writes results to disk, may take several minutes, and triggers a confirm form on clients that support forms. That is meaningful disclosure beyond structured fields.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the highest-stakes facts (cost, paid calls, disk writes, confirmation) before the mechanics. It is dense and the back half is a run-on sequence of parameter roles, but nearly every clause carries distinct information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return values need not be explained. Given a 9-parameter, non-idempotent, costly tool, the description covers cost, duration, disk side effects, prerequisites, confirmation flow, and the cheaper alternative — enough for an agent to decide and call safely.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so the baseline is 3, but the description adds precedence semantics the schema does not state: judgePanel 'overrides judges and judgeModels', turns applies one count to every scenario, and synthesizerModel reconciles judge scores. It does not explain every parameter, but it clarifies the interaction between overlapping ones.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific action and scope: runs 1-4 models across a bench's scenarios and saves a leaderboard. It also distinguishes itself from siblings by naming tournament_quick_test and tournament_plan_run, so an agent can route correctly without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit preconditions (call tournament_plan_run first, get the user's yes in chat) and an explicit alternative for the cheap path (tournament_quick_test). This is exactly the when/when-not/alternative structure the dimension rewards.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

tournament_get_runGet a saved tournament runA

Read-onlyIdempotent

Free local read (no API calls, no network). Reads the full detail of one saved tournament run by ID: plugin, models, judges, scenarios, per-model per-scenario scores, and any failures. Use tournament_leaderboard or the tournament://runs resource to find run IDs.

Name	Required	Description	Default
`runId`	Yes	A run ID, e.g. "run-2026-07-20-235500". Must match a saved run folder.

Output Schema

Name	Required	Description
`run`	Yes

TDQS

A4.3/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint and openWorldHint=false, so safety is covered. The description adds genuinely new context beyond them: 'Free local read (no API calls, no network),' which tells the agent there is no cost or latency risk in calling it repeatedly.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the cost/locality fact before the functional description, then the ID-discovery hint. No filler; every clause carries information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter local read with an output schema already defining the return shape, nothing an agent needs is missing: what it does, that it is free/local, and how to source the required ID. Listing the returned fields is mildly redundant with the output schema but reinforces scope rather than crowding it.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the single runId parameter is already documented with an example format ('run-2026-07-20-235500') and the saved-run-folder constraint. The description only restates 'by ID' and adds no syntax or lookup detail beyond the schema, so the baseline of 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: reads the full detail of one saved tournament run by ID, and enumerates what that detail includes (plugin, models, judges, scenarios, scores, failures). It distinguishes itself from siblings by pointing to tournament_leaderboard and the tournament://runs resource as ID-discovery tools rather than rivals.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear context for use: retrieve one run's full detail by ID, and directs the agent to tournament_leaderboard or the tournament://runs resource to obtain that ID. It stops short of an explicit when-not-to-use rule (e.g., use the leaderboard when you only need aggregate ranking), so it is strong but not exhaustive.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

tournament_leaderboardRead the model leaderboardA

Read-onlyIdempotent

Free local read (no API calls, no network). Reads the best score per model across all saved tournament runs on disk, sorted highest first. Optionally filter to one bench with plugin. Returns an empty list if no runs have been saved yet; run tournament_quick_test to create one.

Name	Required	Description	Default
`limit`	No	Maximum number of models to return, 1-50. Defaults to 10.
`plugin`	No	Only include runs for this bench, e.g. "dnd" or "coding". Omit for all benches.

Output Schema

Name	Required	Description
`entries`	Yes

TDQS

A4.3/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly/idempotent/openWorld=false, but the description adds genuinely new behavior: no API calls or network, highest-first ordering, and an empty-list result when nothing is saved. It stops short of describing tie-breaking or dedup semantics across runs.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences with the cost profile front-loaded, then behavior, then the empty-state escape hatch. No filler and no redundancy.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return-value shape needn't be explained, and the description still covers the cost profile, ordering, filtering, and the zero-run case. Nothing an agent needs to call this correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 100%, so both parameters are already documented in the schema; baseline 3 applies. The description restates the `plugin` filter's purpose but adds no format detail beyond the schema, and never mentions `limit`.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (reads) and resource (best score per model across all saved tournament runs), plus the sort order. It also names a sibling (tournament_quick_test) in the empty-state case, so an agent can place it relative to the rest of the tournament family.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives clear context for when the tool applies (reading the aggregated leaderboard) and routes the agent to tournament_quick_test when no runs exist. It does not contrast itself with tournament_get_run or tournament_list_benches, so the alternative-selection guidance is incomplete.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

tournament_list_benchesList evaluation benchesA

Read-onlyIdempotent

Free local read (no API calls, no network). Lists every registered evaluation bench (plugin) and its scenarios, with IDs and descriptions. Call this first to discover valid plugin and scenario IDs before calling tournament_evaluate or tournament_quick_test.

Name	Required	Description	Default
No parameters

Output Schema

Name	Required	Description
`benches`	Yes

TDQS

A4.7/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, and openWorldHint=false. The description adds that this is 'Free local read (no API calls, no network)', which reinforces the local/no-cost behavior and is useful context. It doesn't explain output structure, but an output schema exists, so that burden is offloaded.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the read nature and scope, then the routing instruction. No wasted text.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present and zero params, the description provides everything needed: what it does, that it's safe/local, and exactly when to call it relative to siblings.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Zero parameters, so baseline is 4. The description correctly implies no input is needed and explains the output fields (IDs and descriptions) that callers will consume.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

Specific verb+resource ('Lists every registered evaluation bench and its scenarios') plus explicit return contents (IDs and descriptions). It distinguishes itself from siblings by naming tournament_evaluate and tournament_quick_test as the consumers of its output.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly states 'Call this first to discover valid plugin and scenario IDs before calling tournament_evaluate or tournament_quick_test.' This gives a clear when-to-use and names the alternatives it feeds.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

tournament_optionsExplore tournament optionsA

Read-onlyIdempotent

Free discovery. Lists benches, scenarios, model tiers with prices per million tokens, judge personas, provider setup, defaults, and limits. Fetches the model catalog but makes no paid model calls.

Name	Required	Description	Default
`bench`	No	Show scenarios for just this bench. Omit for all benches.

Output Schema

Name	Required	Description
`limits`	Yes
`models`	Yes
`benches`	Yes
`defaults`	Yes
`personas`	Yes
`providers`	Yes

TDQS

A3.7/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, and openWorldHint, so the safety profile is covered. The description adds genuinely useful cost context beyond that: the call is free and makes no paid model calls, which is the key behavioral fact for a catalog tool in a paid-model product. It stops short of describing size or pagination of the catalog.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the 'Free discovery' framing and the no-paid-calls guarantee, then the inventory. The long comma list is dense but each item earns its place by telling the agent what the catalog contains.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

An output schema exists, so return-value detail is not required, and annotations cover the safety semantics. For a zero-required-parameter discovery tool, the description is complete enough to call correctly; only an explicit pointer to the next-step sibling is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the single optional 'bench' parameter is fully documented in the schema, including the 'omit for all benches' behavior. The description adds no syntax, format, or naming detail for the parameter, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (lists) plus a concrete inventory of resources: benches, scenarios, model tiers with prices, judge personas, provider setup, defaults, and limits. It also clarifies the tool fetches the catalog without paid calls. It does not explicitly name which sibling to use instead, but the discovery-vs-execution split against tournament_plan_run/tournament_evaluate is inferable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'Free discovery' implies this is the entry point for exploration before committing to a paid run, and 'Fetches the model catalog but makes no paid model calls' reinforces the safe/browse context. However, no explicit when-to-use, when-not-to-use, or named alternatives are given, so the agent must infer that this precedes tournament_plan_run.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

tournament_plan_runPreview a tournament runA

Read-onlyIdempotent

Free preview. Checks your choices against the model catalog, fills in defaults, and shows the plan, rough cost, and any provider setup still needed. Makes no model calls and creates no run folder. Show the preview and get the user's yes before tournament_evaluate.

Name	Required	Description	Default
`bench`	No	Bench name. Defaults to "dnd". Call tournament_options for choices.	dnd
`turns`	No	1-10 turns for every scenario. Omit to use each scenario's own default.
`scenarios`	No	Scenario IDs. Omit for every scenario in the bench.
`candidates`	Yes	1-4 model refs. Bare IDs use OpenRouter. Use anthropic:claude-... for your Anthropic API key. Add @level to set how hard a model thinks, e.g. "openai/gpt-6.1-sol@high". Levels: none, minimal, low, medium, high, xhigh, max; each model accepts only some (see tournament_options). Leave it off for the model's default. The same model may appear twice at different levels.
`judgePanel`	No	1-5 judge seats. Each may set model and either persona or customPersona with a lens (1-1000 characters) and optional name (1-60 characters). Add @level to set how hard a model thinks, e.g. "openai/gpt-6.1-sol@high". Levels: none, minimal, low, medium, high, xhigh, max; each model accepts only some (see tournament_options). Leave it off for the model's default. The same model may appear twice at different levels.
`participant`	No	Model ref for the simulated user who sends follow-up messages. Omit for the default. Add @level to set how hard a model thinks, e.g. "openai/gpt-6.1-sol@high". Levels: none, minimal, low, medium, high, xhigh, max; each model accepts only some (see tournament_options). Leave it off for the model's default. The same model may appear twice at different levels.
`synthesizer`	No	Model ref used to reconcile judge scores. Omit for the default. Not used for one judge. Add @level to set how hard a model thinks, e.g. "openai/gpt-6.1-sol@high". Levels: none, minimal, low, medium, high, xhigh, max; each model accepts only some (see tournament_options). Leave it off for the model's default. The same model may appear twice at different levels.

Output Schema

Name	Required	Description
`plan`	Yes
`summary`	Yes
`estimate`	Yes
`warnings`	Yes
`readyToRun`	Yes

TDQS

A4.4/5.0

Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, and openWorldHint, so the safety profile is known. The description nevertheless adds specific value beyond them: 'Makes no model calls and creates no run folder' and 'Free preview', which tell the agent the cost and side-effect surface more concretely than the generic hints do. It stops short of describing the response shape, but an output schema exists.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with 'Free preview.' and three dense sentences that each carry weight: the no-side-effects constraint, the validation/defaults/plan behavior, and the workflow instruction. No filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 7-parameter tool with full schema coverage, annotations, and an output schema, the description covers purpose, side effects, cost, and workflow handoff adequately. Only minor gaps remain in describing what the preview output looks like, which the output schema likely handles.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so every parameter (bench, turns, scenarios, candidates, judgePanel, participant, synthesizer) is already documented with defaults, ranges, and the @level syntax. The description adds only the general notion that defaults get filled in, not per-parameter semantics. Baseline 3 is correct here.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('preview a tournament run') and enumerates what the preview does: validate against the model catalog, fill defaults, show plan, rough cost, and remaining provider setup. It cleanly separates itself from tournament_evaluate, the sibling that actually runs the tournament.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly sequences the workflow: run this preview, show it, get the user's 'yes', and only then call tournament_evaluate. That is a concrete when-to-use rule tied to a named alternative, not implied guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

tournament_quick_testQuick sanity-check a modelA

The cheap check: one model, one scenario, one judge. Makes real, paid model calls and writes a result to disk. OpenRouter is the default. An anthropic: ref bills your Anthropic API key. Optional judge picks the judge model and persona; turns sets 1-10 turns. No confirm form. Call tournament_options if you need bench or scenario IDs.

Name	Required	Description	Default
`judge`	No	One judge seat with an optional model and either persona or customPersona (lens and optional name). Add @level to set how hard a model thinks, e.g. "openai/gpt-6.1-sol@high". Levels: none, minimal, low, medium, high, xhigh, max; each model accepts only some (see tournament_options). Leave it off for the model's default. The same model may appear twice at different levels.
`model`	Yes	Model ref, e.g. "deepseek/deepseek-v3.2" or "anthropic:claude-haiku-4-5". Add @level to set how hard a model thinks, e.g. "openai/gpt-6.1-sol@high". Levels: none, minimal, low, medium, high, xhigh, max; each model accepts only some (see tournament_options). Leave it off for the model's default. The same model may appear twice at different levels.
`turns`	No	1-10 turns for every scenario. Omit to use each scenario's own default.
`plugin`	No	Bench to test against, e.g. "dnd" or "coding". Defaults to "dnd".	dnd
`scenario`	No	Scenario ID within the plugin. Defaults to the plugin's first scenario.

Output Schema

Name	Required	Description
`runId`	Yes
`judges`	Yes
`plugin`	Yes
`status`	Yes
`entries`	Yes
`message`	No
`failures`	Yes
`resultsDir`	Yes
`judgeFailures`	Yes

TDQS

A4.4/5.0

Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations only declare readOnlyHint=false, openWorldHint=true, idempotentHint=false and destructiveHint=false; the description goes well beyond that by disclosing that calls are real and paid, that a result is written to disk, which provider is billed for an 'anthropic:' ref, which provider is the default, and that there is no confirmation step. These are exactly the operational facts an agent needs before triggering a spend.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Short declarative sentences, front-loaded with the defining scope ('one model, one scenario, one judge') and the cost/write warning immediately after. No filler, and the sibling pointer is correctly placed last as an escape hatch.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With an output schema present, return values need no explanation, and the description covers the non-obvious risks: paid calls, disk writes, billing routing, defaults, and the absence of a confirm form. Only a stated alternative for heavier runs is missing, which is minor.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema already documents model refs, @level suffixes, judge shape, turns range, and plugin defaults in more detail than the description. The description restates 'Optional judge' and 'turns sets 1-10 turns' but adds no syntax or fallback meaning beyond the schema, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The opening 'The cheap check: one model, one scenario, one judge' states a concrete verb (check/test) and scopes the resource tightly, and the closing pointer to tournament_options for bench/scenario IDs separates it from a sibling by function. An agent can distinguish this from tournament_plan_run or tournament_evaluate without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'The cheap check' frames the intended use case (quick sanity check rather than a full run), and it explicitly routes to tournament_options when bench or scenario IDs are needed. It stops short of naming a when-not condition or pointing at the alternative for deeper runs, so it earns a 4 rather than a 5.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

11 tool updatesOct 17, 2026
- Addedtournament_create_bench
- Addedtournament_evaluate
- Addedtournament_get_run
- Addedtournament_leaderboard
- Addedtournament_list_benches
- Addedtournament_options
- Addedtournament_plan_run
- Addedtournament_quick_test
- Removedtournament.evaluate
- Removedtournament.leaderboard
- Removedtournament.quick_test
3 tool updatesv0.1.0Jul 26, 2026
- First observedtournament.evaluate
- First observedtournament.leaderboard
- First observedtournament.quick_test

Deploy Server

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

A4.2/5.0

Scored 2026-10-17 14:00 across 8 tools

Disambiguation4/5

Most tools have clearly distinct roles (quick_test vs evaluate by cost/scale, leaderboard vs get_run by aggregate vs single). However, tournament_list_benches and tournament_options overlap on bench/scenario discovery, and plan_run vs evaluate require reading to distinguish preview from execution.

Naming Consistency4/5

Names follow a consistent tournament_ prefix with snake_case. Most use verb_noun (list_benches, plan_run, get_run, create_bench), but tournament_options and tournament_leaderboard are noun-only, a minor deviation from the pattern.

Tool Count5/5

Eight tools is well-scoped for a tournament/evaluation server. Each tool earns its place: discovery, planning, cheap vs full execution, results reading, and bench creation are all distinct responsibilities.

Completeness4/5

The evaluation lifecycle is well covered: discover, plan, execute (cheap and full), read results, and create custom benches, with paid-call safeguards. Minor gaps exist around managing saved artifacts (update/delete benches or runs) and cancelling long-running evaluations.

Maintenance

ActivityMaintained

ResponsivenessNo issues

Resources

Related MCP Connectors

mcp AOAuth
io.scorecard
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
RAG Systems Monitoring Testing & QA Tools
3
https://mcp.scorecard.io/mcp
rubrkit B
com.rubrkit
MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.
AI & Machine Learning Testing & QA Tools
412
https://www.rubrkit.com/api/v1/mcp
anotherai
dev.anotherai
MCP server for building and testing AI agents with multi-model experimentation and insights.
Agent Orchestration Autonomous Agents Testing & QA Tools
2
https://api.anotherai.dev/mcp
Eval Engine API A
io.github.invertedhoologan
Pay-per-call AI evaluation MCP server. Score LLM outputs against benchmark rubrics via Workers AI.
AI & Machine Learning Testing & QA Tools
2
https://eval.zuluworksai.com/mcp

Related MCP Servers

Patronus MCP Serverofficial
AI & Machine Learning Testing & QA Tools
patronus-ai
A
license
Not graded
quality
D
maintenance
Enables running LLM evaluations, experiments, and custom evaluators through a standardized MCP interface.
Updated 2 years ago (2025-03-26 12:33 UTC)
16
Apache 2.0
metrillm-mcp
MetriLLM
A
license
Not graded
quality
C
maintenance
Enables benchmarking of local LLM models (performance and quality) and sharing results to a public leaderboard via MCP tools.
Updated 5 months ago (2026-06-02 14:32 UTC)
10 npm
8
Apache 2.0
genpark-multi-model-prompt
AI & Machine Learning Testing & QA Tools
alphaparkinc
F
license
Not graded
quality
B
maintenance
Enables running deterministic multi-model prompt regression and golden dataset benchmark suites, exposing structured results and telemetry through the Model Context Protocol for integration with MCP-compliant clients.
Updated 2 months ago (2026-09-02 10:53 UTC)
8
-
llm-benchmark-manager
AI & Machine Learning Testing & QA Tools
telferi
A
license
Not graded
quality
A
maintenance
Enables provider-agnostic LLM discovery, capability testing, stability screening, and AIPerf benchmarking through MCP, with persistent SQLite history.
Updated a month ago (2026-09-25 22:07 UTC)
37 PyPI
MIT

mcp-tournament

The flagship study

A model choice you can explain

How it works

Why multi-judge?

When to use this (and when not to)

Quick start

As a local app (BYOK GUI)

As a desktop app (Windows, unsigned preview)

As an MCP server (Claude Desktop, Cursor, Windsurf)

As a CLI

Swapping models, judges, scenarios, and turns

Reasoning levels

Provider accounts

Results viewer

Model routing

Environment variables

How it's tested

Roadmap

Provenance

Contributing

License

Available Tools

Output Schema

Output Schema

Output Schema

Output Schema

Output Schema

Output Schema

Output Schema

Output Schema

Tool Schema Changelog

Maintenance

Resources

Related MCP Connectors

Related MCP Servers

Patronus MCP Serverofficial