mcp-tournament

Build a custom LLM benchmark in a form, run it from a local GUI, MCP client, or CLI, and turn independent judge opinions into ranked, auditable results.

▶ Live demo: watch a real business-strategy run assemble itself, no key needed.

Why this is interesting:

• Disagreement is data: multiple specialist judges score independently; the arbiter preserves outliers and explains where they diverged.

• Benches are declarative: anyone can define scenarios and criteria as JSON or build them in a form, no pipeline code required.

• BYOK and local-first: bring one OpenRouter key, keep the GUI on your machine, and run budget-tier tournaments for cents.

How it works

flowchart LR
    A["Scenario + criteria<br/>plugin / bench JSON"] --> B["EXECUTE<br/>candidate + tool calls"]
    B --> C["JUDGE<br/>N specialists in parallel"]
    C --> D["SYNTHESIZE<br/>merge + flag outliers<br/>never scores independently"]
    D --> E["AGGREGATE<br/>leaderboard + JSON audit trail"]

Three entry points feed the same pipeline:

• GUI: build benches, launch runs, and inspect results locally.

• MCP client: evaluate models from Claude Desktop, Cursor, or Windsurf.

• CLI: script runs, serve MCP over stdio, or print the leaderboard.

Domain logic is pluggable; the pipeline is not. Benches are declarative plugins: a JSON file (or the Build Bench form) defines scenarios, rounds, an optional simulated participant persona, and judging criteria. Code plugins can go further with custom tools; see docs/PLUGINS.md.

Related MCP server: Patronus MCP Server

Why multi-judge?

Single evaluators miss things. A Rules judge catches mechanical errors; a Creative judge catches boring output; a Holistic judge catches "would I keep using this?" The synthesizer never scores independently: it arbitrates, flags outlier judges, and records why they disagreed. Judge disagreements are first-class data, rendered in the viewer:

When to use this (and when not to)

Those tools are better at what they do; this one is for judgment-heavy, domain-specific evals where a single aggregate score hides the story.

Quick start

git clone https://github.com/samalbanese/mcp-tournament.git
cd mcp-tournament
npm run setup                          # installs + builds server and GUI
export OPENROUTER_API_KEY=sk-or-...    # one key, every role

Or skip local setup entirely:

As a local app (BYOK GUI)

node dist/cli.js gui              # http://localhost:4600

Paste your OpenRouter key in Settings (stored in your browser, sent only to this local server, never written to disk), then set your model routing right below it (default candidates from the live catalog with prices, plus the model behind each judge and the synthesizer) and start a run from NEW RUN. BUILD BENCH creates a new benchmark from a form (question, rounds, persona, judging criteria, with an AI-suggest button) and saves it as a JSON plugin, live immediately.

As a desktop app (Windows, unsigned preview)

The same server + GUI wrapped in an Electron window, with the API key stored via OS-level encryption (safeStorage) instead of the browser:

npm --prefix electron install
npm --prefix electron run dist   # unsigned NSIS installer + portable exe → electron/dist-app/

Builds are unsigned for now, so Windows SmartScreen will warn on first run; see electron/README.md.

As an MCP server (Claude Desktop, Cursor, Windsurf)

{
  "mcpServers": {
    "tournament": {
      "command": "node",
      "args": ["<path-to-repo>/dist/index.js"],
      "env": { "OPENROUTER_API_KEY": "sk-or-..." }
    }
  }
}

As a CLI

# The demo: 3 cheap models, 1 bench scenario, 3 judges (~a few cents)
node dist/cli.js run --plugin business-strategy \
  --models "deepseek/deepseek-v3.2,google/gemini-2.5-flash-lite,meta-llama/llama-4-scout" \
  --scenario pricing-pivot --judges 3

# Or the tool-calling showcase: D&D DM with dice/damage tools and an LLM player
node dist/cli.js run --plugin dnd --models "deepseek/deepseek-v3.2" \
  --scenario dnd-combat --judges 3

node dist/cli.js leaderboard
node dist/cli.js serve          # MCP stdio server

Results viewer

gui/ is a self-contained Vite + React static site with no backend; it deploys to any static host (Cloudflare Pages works as-is). It reads committed run JSON and renders rankings, per-judge breakdowns, disagreement callouts, and full transcripts with tool-call inspection.

cd gui && npm install
npm run import-run -- ../results/<runId>   # copy a run into the viewer
npm run build && npm run preview

Model routing

Every role (the candidates, each judge, the synthesizer, the participant agent) is independently model-selectable and routes through OpenRouter by default. One key, any model, no paid first-party API in the demo path. Defaults are all budget-tier (DeepSeek, Qwen Flash, Gemini Flash Lite; a full run costs cents); override per role:

TOURNAMENT_MODEL_JUDGE_RULES=openai/gpt-5.4-mini
TOURNAMENT_MODEL_SYNTHESIZER=moonshotai/kimi-k2.5
TOURNAMENT_MODEL_PARTICIPANT=deepseek/deepseek-v3.2

The routing layer resolves a pluggable ModelClient per role (src/clients/types.ts). That registry is the documented extension point for a Claude Agent SDK route, which authenticates against a local claude /login session so Claude-judged runs draw on a Max/Pro subscription instead of the metered API: the original oracle-tournament design. Two regression tests guard the default: the demo path never resolves to the paid Anthropic API, and the MCP server's logger stays on stderr (stdout is reserved for JSON-RPC).

Environment variables

How it's tested

npm test runs 32 unit tests with no API key required. Two of them are regression guards with a story:

• The MCP logger writes to stderr only. stdout is reserved for JSON-RPC: one stray console.log corrupts the protocol stream and silently breaks every connected MCP client. The guard makes that a failing test instead of a mystery bug report.

• The default route can never resolve to a paid first-party API. The demo path stays BYOK-through-OpenRouter at budget-tier prices; a config regression that would quietly bill someone's Anthropic key fails CI.

An e2e suite (npm run test:e2e) exercises real model calls when a key is present. CI runs build + unit tests + the GUI build on every push and PR.

Roadmap

Deferred deliberately: tournament.compare / report / plugins / scenarios / judges tools, plugin auto-discovery, npm publish, and MCP registry submission.

Provenance

Generalized from oracle-tournament, a D&D-specific model evaluator whose pipeline proved out the multi-judge + arbiter design; this repo makes the domain pluggable.

Contributing

Issues and PRs welcome; the easiest contribution is a new bench JSON. See CONTRIBUTING.md.

License

MIT