What can you do with this server?

This server runs multi-model LLM tournaments with judge panels and ranked, auditable results. * Evaluate models: run 1–4 candidate models through scenarios with a configurable judge panel (1–5 judges) via tournament.evaluate. * Quick smoke test: score a single model on one scenario with one judge and no synthesis step via tournament.quick_test. * View leaderboards: read the best cached score per model from prior result files via tournament.leaderboard. * Choose plugins/scenarios: select domain benches like dnd, business-strategy, or others; filter by specific scenarios. * Configure roles via environment: override judge, synthesizer, and participant models per role. * Run through MCP clients: connect from Claude Desktop, Cursor, Windsurf, or CLI stdio.

How do I use mcp-tournament?

1. Click on "Deploy Server". 2. Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state. 3. In the chat, type @ followed by the MCP server name and your instructions, e.g., "@mcp-tournament Run the business-strategy benchmark on my model" That's it! The server will respond to your query, and you can continue using it as needed. Here is a step-by-step guide with screenshots.

mcp-tournament

by samalbanese

Overview Schema Source Related Servers Discussions Admin

TypeScript

Remote

mcp-tournament

License: MIT TypeScript Node >= 20 MCP server

Build a custom LLM benchmark in a form, run it from a local GUI, MCP client, or CLI, and turn independent judge opinions into ranked, auditable results.

Evaluation workspace with recorded model rankings and judge disagreement

Explore the interactive demo: compare a recorded business-strategy experiment, inspect the evidence, and try your own criterion weights. No key needed.

Why this is interesting:

Disagreement is data: multiple specialist judges score independently; the arbiter preserves outliers and explains where they diverged.
Benches are declarative: anyone can define scenarios and criteria as JSON or build them in a form, no pipeline code required.
BYOK and local-first: bring one OpenRouter key, keep the GUI on your machine, and run budget-tier tournaments for cents.

A model choice you can explain

The evaluation studio turns the recorded pipeline output into a decision workflow:

Results overview: the original leaderboard, criterion comparisons, and material judge dissent in one place.
Compare evidence: choose up to three candidates and inspect their original answers and arbiter assessments side by side.
Decision lab: change the importance of each criterion with sliders or presets, then see the weighted ranking respond immediately.
Export a report: download original scores, judge identities, dissent, and optional exploratory weights as Markdown.
Follow the evidence: open any model's scorecard, individual judge matrix, transcript, or animated run replay.

Decision lab with adjustable priorities and explicitly separated recorded scores

The demo contains real, dated recordings across business strategy, customer support, creative writing, and the D&D showcase. It makes no live model calls. Results are scenario-specific observations, not a statistically validated or universal model ranking. The decision lab recomputes scores from final criteria; it never changes recorded results or hides missing evidence. Candidate and judge models may overlap.

Related MCP server: agent-eval-mcp

How it works

flowchart LR
    A["Scenario + criteria<br/>plugin / bench JSON"] --> B["EXECUTE<br/>candidate + tool calls"]
    B --> C["JUDGE<br/>N specialists in parallel"]
    C --> D["SYNTHESIZE<br/>merge + flag outliers<br/>never scores independently"]
    D --> E["AGGREGATE<br/>leaderboard + JSON audit trail"]

Three entry points feed the same pipeline:

GUI: build benches, launch runs, and inspect results locally.
MCP client: evaluate models from Claude Desktop, Cursor, or Windsurf.
CLI: script runs, serve MCP over stdio, or print the leaderboard.

Domain logic is pluggable; the pipeline is not. Benches are declarative plugins: a JSON file (or the Build Bench form) defines scenarios, rounds, an optional simulated participant persona, and judging criteria. Code plugins can go further with custom tools; see docs/PLUGINS.md.

Plugin	Domain	Kind
`business-strategy`	SMB pricing decision with real numbers to reason about	📄 bench (JSON)
`creative-writing`	Opening chapter + 3 rounds with a developmental-editor persona	📄 bench (JSON)
`customer-support`	Billing dispute with an escalating customer persona	📄 bench (JSON)
`dnd`	Showcase: D&D 5e Dungeon Master with dice/damage tools and an LLM player	⚙️ code plugin
`coding`	Code generation & review	⚙️ code plugin
Yours	Build in the GUI (`#/build`), drop a JSON in `benches/`, or write TypeScript	🛠 you

Why multi-judge?

Single evaluators miss things. A Rules judge catches mechanical errors; a Creative judge catches boring output; a Holistic judge catches "would I keep using this?" The synthesizer never scores independently: it arbitrates, flags outlier judges, and records why they disagreed. Judge disagreements are first-class data, rendered in the viewer:

Model scorecard with judge disagreements

When to use this (and when not to)

You want	Reach for
CI-style assertions and regression gates over prompts at scale	promptfoo
Standardized academic benchmarks (MMLU, HellaSwag, …)	lm-eval-harness
Rubric-scored comparisons on your own scenarios (multi-round conversations, personas, tool use) with judge disagreement preserved instead of averaged away	mcp-tournament

Those tools are better at what they do; this one is for judgment-heavy, domain-specific evals where a single aggregate score hides the story.

Quick start

git clone https://github.com/samalbanese/mcp-tournament.git
cd mcp-tournament
npm run setup                          # installs + builds server and GUI
export OPENROUTER_API_KEY=sk-or-...    # one key, every role

Or skip local setup entirely: Open in GitHub Codespaces

As a local app (BYOK GUI)

node dist/cli.js gui              # http://localhost:4600

Paste your OpenRouter key in Settings (stored in your browser, sent only to this local server, never written to disk), then set your model routing right below it (default candidates from the live catalog with prices, plus the model behind each judge and the synthesizer) and start a run from NEW RUN. BUILD BENCH creates a new benchmark from a form (question, rounds, persona, judging criteria, with an AI-suggest button) and saves it as a JSON plugin, live immediately.

As a desktop app (Windows, unsigned preview)

The same server + GUI wrapped in an Electron window, with the API key stored via OS-level encryption (safeStorage) instead of the browser:

npm --prefix electron install
npm --prefix electron run dist   # unsigned NSIS installer + portable exe → electron/dist-app/

Builds are unsigned for now, so Windows SmartScreen will warn on first run; see electron/README.md.

As an MCP server (Claude Desktop, Cursor, Windsurf)

{
  "mcpServers": {
    "tournament": {
      "command": "node",
      "args": ["<path-to-repo>/dist/index.js"],
      "env": { "OPENROUTER_API_KEY": "sk-or-..." }
    }
  }
}

Tool	Description
`tournament.evaluate`	1–4 models × scenarios × judge panel → ranked results
`tournament.quick_test`	One scenario, one judge: fast smoke score
`tournament.leaderboard`	Best cached score per model across runs

As a CLI

# The demo: 3 cheap models, 1 bench scenario, 3 judges (~a few cents)
node dist/cli.js run --plugin business-strategy \
  --models "deepseek/deepseek-v3.2,google/gemini-2.5-flash-lite,meta-llama/llama-4-scout" \
  --scenario pricing-pivot --judges 3

# Or the tool-calling showcase: D&D DM with dice/damage tools and an LLM player
node dist/cli.js run --plugin dnd --models "deepseek/deepseek-v3.2" \
  --scenario dnd-combat --judges 3

node dist/cli.js leaderboard
node dist/cli.js serve          # MCP stdio server

Results viewer

gui/ is a self-contained Vite + React static site with no backend; it deploys to any static host (Cloudflare Pages works as-is). It reads committed run JSON and renders rankings, per-judge breakdowns, disagreement callouts, and full transcripts with tool-call inspection.

Leaderboard view

cd gui && npm install
npm run import-run -- ../results/<runId>   # copy a run into the viewer
npm run build && npm run preview

Transcript view

Model routing

Every role (the candidates, each judge, the synthesizer, the participant agent) is independently model-selectable and routes through OpenRouter by default. One key, any model, no paid first-party API in the demo path. Defaults are all budget-tier (DeepSeek, Qwen Flash, Gemini Flash Lite; a full run costs cents); override per role:

TOURNAMENT_MODEL_JUDGE_RULES=openai/gpt-5.4-mini
TOURNAMENT_MODEL_SYNTHESIZER=moonshotai/kimi-k2.5
TOURNAMENT_MODEL_PARTICIPANT=deepseek/deepseek-v3.2

The routing layer resolves a pluggable ModelClient per role (src/clients/types.ts). That registry is the documented extension point for a Claude Agent SDK route, which authenticates against a local claude /login session so Claude-judged runs draw on a Max/Pro subscription instead of the metered API: the original oracle-tournament design. Two regression tests guard the default: the demo path never resolves to the paid Anthropic API, and the MCP server's logger stays on stderr (stdout is reserved for JSON-RPC).

Environment variables

Variable	Required	Purpose
`OPENROUTER_API_KEY`	Yes	All roles by default
`TOURNAMENT_MODEL_*`	No	Per-role model overrides (see above)
`TOURNAMENT_RESULTS_DIR`	No	Results output root (default `./results`)

How it's tested

npm run test:unit runs 43 unit tests with no API key required. The suite includes decision-lab tests for changing priorities, zero weights, missing evidence, ties, preserved original scores, and report provenance. The committed demo fixtures are also checked through the viewer loaders, including known partial judge archives. Two additional regression guards have a story:

The MCP logger writes to stderr only. stdout is reserved for JSON-RPC: one stray console.log corrupts the protocol stream and silently breaks every connected MCP client. The guard makes that a failing test instead of a mystery bug report.
The default route can never resolve to a paid first-party API. The demo path stays BYOK-through-OpenRouter at budget-tier prices; a config regression that would quietly bill someone's Anthropic key fails CI.

An e2e suite (npm run test:e2e) exercises real model calls when a key is present. CI runs build + unit tests + the GUI build on every push and PR.

Roadmap

Deferred deliberately: tournament.compare / report / plugins / scenarios / judges tools, plugin auto-discovery, npm publish, and MCP registry submission.

Provenance

Generalized from oracle-tournament, a D&D-specific model evaluator whose pipeline proved out the multi-judge + arbiter design; this repo makes the domain pluggable.

Contributing

Issues and PRs welcome; the easiest contribution is a new bench JSON. See CONTRIBUTING.md.

License

MIT

Available Tools

3 tools

tournament.evaluateB

Evaluate one to four candidate models with a judge panel.

ParametersJSON Schema

Name	Required	Default
`judges`	No
`models`	Yes
`plugin`	No	dnd
`scenarios`	No

TDQS

B3.3/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must cover behavioral aspects. It only states 'evaluate with a judge panel' without disclosing whether the operation is read-only, what side effects occur, authentication needs, or rate limits. This is insufficient for a tool that likely performs comparative analysis.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence that front-loads the verb and resource with no extraneous words. Every part is useful.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has 4 parameters, no output schema, and no parameter descriptions, the description is severely lacking. It does not explain what the evaluation returns, how results are presented, or what 'plugin' and 'scenarios' entail, leaving significant gaps for the AI agent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The input schema has 0% description coverage, and the description adds minimal parameter insight. It mentions 'judge panel' which relates to the 'judges' parameter but does not explain 'plugin' or 'scenarios'. The meaning of parameters beyond their names is largely opaque.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the verb 'evaluate' and the resource 'one to four candidate models' with a 'judge panel'. It distinguishes from sibling tools like quick_test and leaderboard by specifying the use of a judge panel for evaluation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description implies usage for evaluating models with a panel but provides no explicit guidance on when to use this tool versus alternatives like quick_test or leaderboard. No when-not-to-use or prerequisite information is given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

tournament.leaderboardC

Read the best cached score per model from result files.

ParametersJSON Schema

Name	Required	Description	Default
`limit`	No
`plugin`	No

TDQS

C2.6/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

No annotations are provided, so the description must fully disclose behavior. It states 'Read the best cached score' implying a read-only operation, but does not mention potential side effects, authentication requirements, or data source details beyond 'result files'.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence with no unnecessary words. It is well-structured and front-loaded with the key action.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Despite having only 2 parameters and no output schema, the description lacks details about return format, data structure, or how parameters affect results. The 'plugin' parameter is left entirely unexplained.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, and the description does not explain the purpose or valid values of the 'limit' and 'plugin' parameters. The agent cannot infer parameter semantics from the description alone.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool reads cached scores per model from result files, using a specific verb and resource. It distinguishes itself from sibling tools quick_test and evaluate by focusing on cached leaderboard data.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No guidance is given on when to use this tool versus alternatives like tournament.quick_test or tournament.evaluate. The agent has no context about preferred use cases.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

tournament.quick_testC

Run one scenario with one judge and no synthesis model call.

ParametersJSON Schema

Name	Required	Default
`model`	Yes
`plugin`	No	dnd
`scenario`	No

TDQS

C2.6/5.0

Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Without annotations, the description should fully disclose behavioral traits. It mentions specifics like single scenario/judge and no synthesis call, but it does not indicate whether the operation is safe (e.g., read-only), destructive, or has other side effects.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is a single concise sentence with no unnecessary words. It front-loads the core action and key constraints ('one scenario', 'one judge', 'no synthesis model call').

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness1/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

Given the tool has 3 parameters (none with schema descriptions), no output schema, and no annotations, the description is severely incomplete. It fails to explain parameters, return values, or usage context, leaving an agent unable to invoke the tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters1/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must explain parameter meanings. The description only mentions 'one scenario' and 'one judge' but does not map to the parameters 'model', 'plugin', or 'scenario', nor does it clarify their roles or constraints.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool performs a quick test with one scenario, one judge, and no synthesis call. This distinguishes it from sibling tools like tournament.leaderboard and tournament.evaluate, which imply broader or more comprehensive evaluation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No explicit guidance on when to use this tool versus alternatives. The description implies it is for quick tests but does not state prerequisites, limitations, or when not to use it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

3 tool updatesv0.1.0Jul 26, 2026
- First observedtournament.evaluate
- First observedtournament.leaderboard
- First observedtournament.quick_test

Deploy Server

One endpoint for every MCP toolRayrun runs every MCP server behind one URL and holds the credentials. Each call is decided per client, tool, and argument, then recorded.ray.runAd

TDQS

B3/5.0

Scored 2026-07-30 00:35 across 3 tools

Disambiguation5/5

Each tool has a distinct purpose: quick test runs a minimal scenario, leaderboard reads cached scores, and evaluate runs a full evaluation. No overlap or ambiguity.

Naming Consistency2/5

Naming is inconsistent: 'quick_test' uses underscore and adjective+noun, 'leaderboard' is a single noun without underscore, and 'evaluate' is a bare verb. No consistent pattern in structure or part of speech.

Tool Count4/5

With 3 tools, the server is at the low end of the typical 3-15 range but still reasonable for a focused evaluation service. The scope is narrow enough that each tool earns its place.

Completeness4/5

The tools cover the core workflow: quick test, full evaluation, and reading results. Minor gaps exist, such as no tool for configuring judges or scenarios, but these are likely predefined or managed externally.

Maintenance

ActivityMaintained

ResponsivenessNo issues

Resources

GitHub Repository

Need Help?

Related Servers

Related MCP Connectors

mcp AOAuth
io.scorecard
MCP server providing access to the Scorecard API to evaluate and optimize LLM systems.
RAG Systems Monitoring Testing & QA Tools
3
rubrkit B
com.rubrkit
MCP-native AI evaluation: rubric audits, eval suites, and proof reports for AI/LLM output.
AI & Machine Learning Testing & QA Tools
412
anotherai
dev.anotherai
MCP server for building and testing AI agents with multi-model experimentation and insights.
Agent Orchestration Autonomous Agents Testing & QA Tools
2
Eval Engine API A
io.github.invertedhoologan
Pay-per-call AI evaluation MCP server. Score LLM outputs against benchmark rubrics via Workers AI.
AI & Machine Learning Testing & QA Tools
2

Related MCP Servers

Patronus MCP Serverofficial
AI & Machine Learning Testing & QA Tools
patronus-ai
A
license
Not graded
quality
D
maintenance
Enables running LLM evaluations, experiments, and custom evaluators through a standardized MCP interface.
Updated a year ago (2025-03-26 12:33 UTC)
16
Apache 2.0
agent-eval-mcp
Testing & QA Tools CI/CD & DevOps AI & Machine Learning
ad-github1
A
license
B
quality
D
maintenance
An MCP-style stdio server for evaluating AI agent outputs, enabling CI-friendly quality gates, regression comparisons, and canary promotion decisions.
Updated 3 months ago (2026-06-29 10:09 UTC)
3
MIT
AgentOps EvalBench MCP
AI & Machine Learning RAG Systems Observability
AbhinavVarma02
A
license
Not graded
quality
A
maintenance
Enables LLM evaluation and observability by uploading documents, building test sets, running RAG pipelines, and automatically scoring answers for groundedness, hallucination risk, retrieval quality, latency, and cost, with tools exposed to MCP-compatible clients.
Updated 20 days ago (2026-09-01 03:01 UTC)
1
MIT
metrillm-mcp
MetriLLM
A
license
Not graded
quality
C
maintenance
Enables benchmarking of local LLM models (performance and quality) and sharing results to a public leaderboard via MCP tools.
Updated 4 months ago (2026-06-02 14:32 UTC)
15 npm
8
Apache 2.0

mcp-tournament

A model choice you can explain

How it works

Why multi-judge?

When to use this (and when not to)

Quick start

As a local app (BYOK GUI)

As a desktop app (Windows, unsigned preview)

As an MCP server (Claude Desktop, Cursor, Windsurf)

As a CLI

Results viewer

Model routing

Environment variables

How it's tested

Roadmap

Provenance

Contributing

License

Available Tools

TDQS

TDQS

TDQS

Tool Schema Changelog

TDQS

Maintenance

Resources

Related MCP Connectors

Related MCP Servers

Patronus MCP Serverofficial

agent-eval-mcp

AgentOps EvalBench MCP

metrillm-mcp