BlindWrite MCP
You can delegate long-form writing to OpenRouter models and run blind, bias-free benchmarks that rank models based on your preferences.
Create writing benchmark tasks with categories, prompts, and evaluation criteria.
List available models with pricing and enabled status.
Generate anonymous writing outputs from multiple models via OpenRouter.
Start randomized blind A/B duels with model identities masked.
Submit preference votes (A, B, or tie) with optional reasoning and dimension scores.
Retrieve benchmark results, unmasking identities only on explicit reveal.
Get personal or global leaderboards using Bradley-Terry MLE, Elo, or win rate.
Compare two models head-to-head from accumulated battle history.
Analyze empirical writing preferences (conciseness, structure, tone) from votes.
Get detailed performance stats for a model (win rate, latency, cost).
Delegate drafting via
writer_generateto save Claude output tokens, using your top-ranked or specified OpenRouter model.
Allows generating writing drafts and running blind A/B benchmark duels with Google Gemini models via OpenRouter.
Allows generating writing drafts and running blind A/B benchmark duels with Meta Llama models via OpenRouter.
Allows generating writing drafts and running blind A/B benchmark duels with OpenAI GPT-4o via OpenRouter.
Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type
@followed by the MCP server name and your instructions, e.g., "@BlindWrite MCP Run a blind A/B test on my landing page headline between two OpenRouter models."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
BlindWrite MCP ποΈ
Think with Claude, Write with OpenRouter. Production-grade Model Context Protocol (MCP) server for blind, bias-free AI writing benchmarks and high-speed token-saving writing delegation inside Claude Desktop.
Just want to install it? See INSTALL.md for step-by-step setup only.
Custom instructions to prevent Claude bypassing BlindWrite MCP? See CLAUDE_PROMPT.md β paste it into Claude Desktop's Custom Instructions so drafting always delegates to writer_generate, even if skill routing fails.
Looking for the full content production pipeline? Jump to Section 7: Autonomous Content Production Pipeline (/create-content) to produce complete, audited, publish-ready articles with DataForSEO, Firecrawl, and OpenRouter in a single command.
1. Problem & Purpose
Writers, marketers, and knowledge workers face two major challenges when using LLMs for writing:
Token Inefficiency & Cost: Asking Claude to generate thousands of words of draft text burns through Claude output limits and tokens rapidly.
Brand Bias in Model Selection: When picking an AI model, brand names ("GPT-4o", "Claude 3.5 Sonnet") skew perception, even when lightweight, focused models might write better copy for your specific voice.
BlindWrite MCP introduces the "Think with Claude, Write with OpenRouter" hybrid workflow:
Claude for Thinking & Strategy: Claude handles deep reasoning, structural outlines, audience angles, and critical review.
OpenRouter for Heavy Writing: Claude delegates long-form drafting directly via
writer_generateto your top-ranked OpenRouter model (GLM 5.3), generating drafts in seconds while saving thousands of Claude generation tokens.Blind A/B Benchmarking: When testing styles, competing frontier models generate blind drafts (
Response AvsResponse B) with zero brand leakage. You vote, and the Bradley-Terry MLE & Elo engine trains your personal leaderboard!Preference Analytics: Discovers empirical writing metrics (conciseness, bulleted structure, formality) to show why your top models resonate with you.
Related MCP server: panel-review MCP server
2. Architecture
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Claude Desktop (Client UI) β
ββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ
β
β stdio JSON-RPC (MCP)
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β BlindWrite MCP β
β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β MCP Tools Layer β β
β β 10 Tools β’ Zod Schemas β’ Strict Blindness Gating β β
β βββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ β
β β β
β βββββββββββββββββββββββββββββΌββββββββββββββββββββββββββββ β
β β Domain Services Layer β β
β β - BenchmarkService - Duel & RandomizationService β β
β β - VotingService - Bradley-Terry & Elo Engine β β
β β - AnalyticsService - OpenRouterClient & Cost β β
β βββββββββββββββ¬ββββββββββββββββββββββββββββ¬ββββββββββββββ β
β β β β
β βΌ βΌ β
β ββββββββββββββββββββββββββββ ββββββββββββββββββββββββββββ β
β β SQLite Data Layer β β OpenRouter Gateway β β
β β - WAL mode & Pragma FK β β - Latency measurement β β
β β - Repositories: Tasks, β β - Token & cost tracking β β
β β Models, Outputs, β β - Error & retry policy β β
β β Battles, Votes β ββββββββββββββ¬ββββββββββββββ β
β ββββββββββββββββββββββββββββ β β
ββββββββββββββββββββββββββββββββββββββββββββββΌβββββββββββββββββ
β HTTPS
βΌ
βββββββββββββββββββββββββββββββ
β OpenRouter API β
β βββββββββββ¬ββββββββββ¬ββββββ β
β β Claude β GPT-4o β ... β β
β βββββββββββ΄ββββββββββ΄ββββββ β
βββββββββββββββββββββββββββββββCritical Stdio Hygiene: Standard output (
stdout) is strictly reserved for MCP JSON-RPC protocol packets. All internal diagnostics, logging, and error output route exclusively tostderr, guaranteeing that Claude Desktop never crashes from corrupted transport data.
3. The 11 MCP Tools
Tool Name | Purpose | Role in "Think with Claude, Write with OpenRouter" |
| Directly generates drafts with OpenRouter | Token-Saver: Claude outlines; OpenRouter model writes the draft |
| Creates a new writing benchmark task | Assigns clean task ID; status |
| Lists models with pricing & status | Used for model selection and registry status |
| Dispatches prompt to models in parallel | Returns ONLY anonymous tokens ( |
| Starts a randomized, blind A/B battle | 50/50 coin toss determines Response A vs B; eliminates brand bias |
| Submits preference ( | Records vote, updates Bradley-Terry & Elo ratings |
| Retrieves task results; unmasks on reveal | Identities remain hidden until explicit user reveal |
| Returns personal or global leaderboards | Bradley-Terry MLE & Elo rankings guide |
| Head-to-head comparison of two models | Direct win/loss record and dimensional ratings |
| Discovers personal writing style preferences | Explains why certain models work best for you |
| Deep performance card for a single model | Battle record, win rate, latency, and token cost |
4. Pre-Seeded Writing Categories & Models
Writing Categories
General WritingEmails(Cold outreach, executive memos, customer service)Business Writing(Proposals, pitches, quarterly summaries)Creative Writing(Narrative storytelling, dialogue, fiction)Marketing(Ad copy, landing pages, taglines)Summarization(Executive briefs, meeting notes)Rewriting(Tone shifting, simplifying complex text)Technical Writing(API documentation, engineering RFCs)Persuasive Writing(Op-eds, debate arguments, sales letters)Research Writing(Literature synthesis, analytical essays)Social Media(Threads, LinkedIn posts, announcements)Instruction Following(Constraint adherence, exact formatting)
Seed Model (OpenRouter)
GLM 5.3 (
z-ai/glm-5.3)
5. Quickstart & Installation
Option A: 1-Line Autonomous Remote Installer (Fastest β No Git Clone Required)
Run a single command in your terminal. It will automatically check/install Node.js & Git, clone into your user directory, build the project, prompt for your OpenRouter key, auto-install the Writing Orchestrator Skill, and configure Claude Desktop:
Windows (PowerShell):
irm https://raw.githubusercontent.com/ygstudio-game/BlindWrite_MCP/main/scripts/install.ps1 | iex(Tip: You can pre-supply your key to make it 100% headless: $env:OPENROUTER_API_KEY="sk-or-v1-..."; irm https://raw.githubusercontent.com/ygstudio-game/BlindWrite_MCP/main/scripts/install.ps1 | iex)
macOS & Linux (Terminal):
curl -fsSL https://raw.githubusercontent.com/ygstudio-game/BlindWrite_MCP/main/scripts/install.sh | bash(Tip: Or headless: OPENROUTER_API_KEY="sk-or-v1-..." curl -fsSL https://raw.githubusercontent.com/ygstudio-game/BlindWrite_MCP/main/scripts/install.sh | bash)
Manual Skill Installation (If using 1-Line Remote Installer):
While the remote installer automatically registers the MCP server in claude_desktop_config.json, you can also manually install or upload the Writing Orchestrator Skill to your Claude environment:
Claude Desktop App (Account Upload via UI β ZIP or Direct MD):
In Claude Desktop, open Customize > Skills from the sidebar.
Click the
+button and choose "Upload a skill".You can upload either:
ZIP Archive: Select
writing-orchestrator.zip(located at%LOCALAPPDATA%\BlindWrite_MCP\skills\writing-orchestrator.zipon Windows, or~/.blindwrite-mcp/skills/writing-orchestrator.zipon macOS/Linux). If you manually zip the folder on Windows, right-clickwriting-orchestratorfolder β Compress to ZIP file (or use 7-Zip).Direct
.mdFile (No ZIP needed!): Directly selectSKILL.mdfrom yourskills/writing-orchestrator/SKILL.mdfolder.
Ensure the skill toggle is switched ON (synced directly to your Anthropic account in the cloud).
Claude Code / Terminal Integration (Local Filesystem):
Windows (PowerShell):
New-Item -ItemType Directory -Force -Path "$HOME\.claude\skills\writing-orchestrator" Invoke-WebRequest -Uri "https://raw.githubusercontent.com/ygstudio-game/BlindWrite_MCP/main/skills/writing-orchestrator/SKILL.md" -OutFile "$HOME\.claude\skills\writing-orchestrator\SKILL.md"macOS / Linux (Bash):
mkdir -p ~/.claude/skills/writing-orchestrator && curl -fsSL https://raw.githubusercontent.com/ygstudio-game/BlindWrite_MCP/main/skills/writing-orchestrator/SKILL.md -o ~/.claude/skills/writing-orchestrator/SKILL.md
Zero-Setup Native MCP Prompt:
The installer already configures Claude Desktop to expose the
/writing-orchestratorMCP Prompt natively. You can type/writing-orchestratordirectly in any chat without copying any files!
Option B: Local 1-Click Setup (If already cloned)
On Windows:
Simply run setup.bat (or double-click it in File Explorer):
setup.batAuto-installs Node.js via
wingetif not detected on your system.Installs dependencies (
npm install).Compiles TypeScript (
npm run build).Prompts for your OpenRouter API key.
Automatically merges the configuration into
%APPDATA%\Claude\claude_desktop_config.json(with automatic backup).Auto-installs the Writing Orchestrator Skill into
~/.claude/skills/writing-orchestratorand agent environments.
On macOS / Linux:
chmod +x setup.sh
./setup.sh
# or after npm install:
npm run setupOption C: Manual Setup
Clone and Install:
git clone https://github.com/ygstudio-game/BlindWrite_MCP.git
cd BlindWrite_MCP
npm installConfigure Environment:
cp .env.example .envEdit .env and set OPENROUTER_API_KEY=sk-or-v1-....
Build and Run Setup Wizard:
npm run build
npm run setupThe interactive wizard will automatically resolve exact paths and update Claude Desktop for you!
π How to Update BlindWrite MCP
To update your existing installation to the latest version:
Method 1: Re-run the 1-Line Remote Installer (Fastest)
Simply re-run the 1-line command in your terminal. It will automatically detect your existing installation, pull the latest updates from GitHub, and rebuild:
Windows (PowerShell):
irm https://raw.githubusercontent.com/ygstudio-game/BlindWrite_MCP/main/scripts/install.ps1 | iexmacOS / Linux (Bash):
curl -fsSL https://raw.githubusercontent.com/ygstudio-game/BlindWrite_MCP/main/scripts/install.sh | bash
Method 2: Terminal Update
Navigate to your installation directory and pull the latest code:
Windows (PowerShell):
cd "$env:LOCALAPPDATA\BlindWrite_MCP" git fetch origin main git reset --hard origin/main npm install --ignore-scripts npm run build node scripts/setup.js --yesmacOS / Linux (Bash):
cd ~/.blindwrite-mcp git fetch origin main git reset --hard origin/main npm install --ignore-scripts npm run build node scripts/setup.js --yes
(If you cloned into a custom directory, run git pull origin main && npm run build inside that folder).
After Updating:
Restart Claude Desktop: Completely quit Claude Desktop (Right-click tray icon β Quit) and reopen it.
Skill Update (Optional): In Claude Desktop (Customize > Skills), re-upload
writing-orchestrator.ziporSKILL.mdto refresh your account skill.
6. Claude Desktop Configuration (Manual)
If you prefer to configure Claude Desktop manually instead of using npm run setup:
Open your Claude Desktop configuration file:
Windows:
%APPDATA%\Claude\claude_desktop_config.jsonmacOS:
~/Library/Application Support/Claude/claude_desktop_config.json
Add the
blindwriteserver:
{
"mcpServers": {
"blindwrite": {
"command": "node",
"args": [
"/absolute/path/to/BlindWrite_MCP/dist/index.js"
],
"env": {
"OPENROUTER_API_KEY": "sk-or-v1-your-openrouter-key-here",
"DB_PATH": "/absolute/path/to/BlindWrite_MCP/data/blindwrite.sqlite",
"LOG_LEVEL": "info"
}
}
}
}Platform Path Examples:
macOS / Linux:
"/Users/yourname/projects/BlindWrite_MCP/dist/index.js"Windows:
"C:\\Users\\yourname\\projects\\BlindWrite_MCP\\dist\\index.js"(ensure backslashes are escaped with\\)
Restart Claude Desktop. You will see the hammer icon with all 11 BlindWrite tools ready to use!
Writing Orchestrator Skill & Prompt:
Skill Source: Read the complete skill definition in
skills/writing-orchestrator/SKILL.md.Upload to Claude Desktop: In Claude Desktop (Customize > Skills >
+> Upload a skill), upload eitherwriting-orchestrator.zipor selectSKILL.mddirectly (no zip required).Manual ZIP Creation: If manually zipping the folder on Windows, right-click
writing-orchestratorfolder β Compress to ZIP file (or use 7-Zip).Auto-Installed Local Folders: The setup script auto-populates
%APPDATA%\Claude\skills\writing-orchestrator\SKILL.md,~/.claude/skills/writing-orchestrator/SKILL.md, and project-level.claude/skills/.Native MCP Prompt: The server natively registers the
/writing-orchestratorprompt accessible directly inside Claude Desktop chat without uploading any files!Project Instructions: For Claude Desktop Projects, copy
CLAUDE_PROMPT.mdinto your Project's Custom Instructions.How to invoke in chat: Simply tell Claude:
"Use the writing-orchestrator skill to outline our strategy and draft..."
or select the/writing-orchestratorprompt template from the chat input menu.
7. Autonomous Content Production Pipeline (/create-content)
One Invocation. One Publish-Ready Article.
The autonomous content pipeline orchestrates Claude, DataForSEO, Firecrawl, and OpenRouter into an uninterrupted end-to-end publishing pipeline. It handles preflight checks, intake, SERP/competitor research, quality checklist compilation, high-speed drafting, dual-gate AI/EEAT auditing, and automated revision loops with zero manual glue.
flowchart TD
subgraph Phase0 [Phase 0: Preflight]
P0A[Verify MCP Bridge] --> P0B[Resolve Site from Registry]
P0B --> P0C[Verify Voice Doc]
P0C --> P0D[Confirm Enabled Models]
end
subgraph Phase1 [Phase 1: Intake]
P1A[Capture Topic & Constraints] --> P1B[Generate Job ID]
P1B --> P1C[Load Content Profile]
end
subgraph Phase2 [Phase 2: Research & Brief]
P2A[DataForSEO SERP Analysis] --> P2B[Firecrawl Competitor Extraction]
P2B --> P2C[Compile MASTER_WRITING_BRIEF]
end
subgraph Phase3 [Phase 3: Checklist & Voice]
P3A[Verbatim Quality Checklist] --> P3B[Verbatim Voice Requirements]
P3B --> P3C[Assemble Prompt Architecture]
end
subgraph Phase4 [Phase 4: Draft & Revision Loop]
P4A[OpenRouter Drafting via writing-orchestrator] --> P4B[Claude Dual-Gate Audit]
P4B -->|Needs Revision max 3x| P4C[Targeted Fix Prompt]
P4C --> P4A
end
subgraph Phase5 [Phase 5: Clean Delivery]
P4B -->|Pass| P5A[Save to data/drafts/]
P5A --> P5B[Log Job to data/jobs/]
P5B --> P5C[Clean Markdown Output in Chat]
end
Phase0 --> Phase1 --> Phase2 --> Phase3 --> Phase4 --> Phase57.1 How to Run (Single Invocation)
In Claude Desktop or Claude Code, simply invoke /create-content:
/create-content
Site: example.com
Keyword: productivity tools for remote teams
Topic: Complete Guide to Remote Team Productivity
Content type: seo_article
Target length: 1500-2000 words
Special requirements: Include comparison table of key features and workflows(Note: If your active Claude Project name matches the site, or if you omit Site:, the universal default profile is applied automatically).
7.2 The 6 Execution Phases
Phase | Responsibility | What Happens Autonomously |
Phase 0: Preflight | 4 Hard Stops | 1. Confirms BlindWrite MCP is online via |
Phase 1: Intake | Task Framing | Assigns a unique |
Phase 2: Pre-Writing Research | Live Data Grounding |
|
Phase 3: Checklist & Voice Layer | Prompt Compilation | Copies verbatim rules from |
Phase 4: Draft & Grading Loop | OpenRouter Execution + Claude Audit | Calls |
Phase 5: Delivery & Logging | Clean Publishing | Auto-saves the finalized markdown to |
7.3 Site Configuration & Profiles
The create-content skill is 100% self-contained and works for any website or niche out of the box with built-in default and customizable profiles:
site: example.com
status: active # active | out_of_scope | blocked
voice_doc_path: "d:/path/to/voice-guidelines.md" # or inline voice guidelines
persona: "Senior Tech Journalist & Product Specialist"
hard_rules:
- "Cite verifiable benchmark and workflow data."
- "No artificial hype or marketing fluff."
content_profiles:
seo_article:
pre_writing_skill: semantic-seo-content
voice_skill: heading-microcopy-writer
target_length_default: "1800-2400 words"7.4 Supported Drafting Model & Cost Breakdown
The pipeline exclusively uses GLM 5.3 configured for high-velocity drafting:
Model Name | OpenRouter ID | Pricing (Input / Output per 1M) | Best For |
GLM 5.3 |
| $0.936 / $3.168 | High Quality: Complex technical synthesis, nuanced argumentation, and high-depth drafting. |
7.5 Stop Conditions & Escalation Handling
The pipeline stops and asks for human intervention only on critical edge cases:
MCP Bridge Disconnected: If
benchmark_list_modelsfails, it halts immediately without silent Claude fallback.Voice Document Missing / "PENDING": Halts to prevent generating content in an uncalibrated tone. Users can provide the file path or explicitly sign off on proceeding with generic voice.
Out of Scope Site: Sites marked
status: out_of_scopeare rejected with the recorded reason.Escalation after 3 Loops: If an audit issue persists across 3 OpenRouter revisions, the pipeline offers three explicit options:
Ship with the audit caveat noted.
One-time exception: Claude directly polishes the remaining flagged lines.
Re-draft with GLM 5.3.
8. Example Workflows in Claude Desktop
Workflow A: High-Conversion B2B Cold Outreach Email
Goal: Craft a personalized, high-response email while saving Claude output tokens.
You:
"Help me draft a 120-word cold outreach email to a VP of Sales about our developer platform. Use the writing-orchestrator skill to outline the strategy first and use writer_generate to draft it."
Claude (Thinking & Strategy Brain):
Analyzes the VP of Sales persona (time-poor, quota-focused, values proven ROI).
Formulates the prompt blueprint (Hook: shortening deal cycles by 40%; Body: 2 concrete metrics; CTA: 10-minute intro).
Delegates drafting: Calls
writer_generate(category: "Emails", prompt: "...", max_tokens: 400).OpenRouter generates the draft in ~800ms using your personal #1 model (GLM 5.3).
Zero-Tax Direct Delivery: Claude presents the generated email verbatim with the metrics badge (saved ~300 Claude output tokens, cost $0.0003), auto-saved to
data/drafts/, without burning extra tokens on unprompted critique.
Workflow B: Technical Architecture RFC / Migration Spec
Goal: Generate a comprehensive 1,500-word engineering RFC with deep strategic framing.
You:
"We need an RFC for migrating our monolithic PostgreSQL database to a globally distributed database with zero downtime. Use the writing-orchestrator skill to structure the technical plan and delegate the drafting."
Claude (Thinking & Strategy Brain):
Produces an architectural breakdown:
Problem statement & current bottlenecks
Dual-write replication architecture & cutover sequence
Failure modes, rollback triggers, and data validation
Calls
writer_generate(category: "Technical Writing", prompt: "Write section 2 and 3 covering dual-write synchronization and consistency models...", max_tokens: 2500).OpenRouter streams the extensive technical draft in seconds.
Claude critically audits the draft, flags edge cases in network partition scenarios, and polishes the summary.
Workflow C: High-Converting SaaS Landing Page Copy
Goal: Draft hero headlines, subheads, and 3 value pillars.
You:
"Write landing page hero section copy and 3 core value pillars for an AI agent observability tool. Use the writing-orchestrator skill."
Claude (Thinking & Strategy Brain):
Outlines the positioning: Anti-hype, focused on debugging production hallucinations and latency spikes.
Calls
writer_generate(category: "Marketing Copy", prompt: "Craft punchy H1 headline, 2-sentence subhead, and 3 customer-outcome pillars with proof points...", max_tokens: 800).Displays the full landing page copy with clear visual hierarchy, ready for deployment.
Workflow D: Thought Leadership / Technical Blog Post
Goal: Write an engaging 800-word essay with a contrarian engineering thesis.
You:
"Draft an engineering blog post arguing why autonomous agent swarms will replace traditional CI/CD pipelines by 2027. Use the writing-orchestrator skill."
Claude (Thinking & Strategy Brain):
Establishes the contrarian hook, 3 narrative beats, and concrete code pipeline analogies.
Calls
writer_generate(category: "Blog Posts", prompt: "...", max_tokens: 1500).OpenRouter drafts the essay using your preferred style profile.
Claude reviews flow, rhythm, and tone, delivering an impactful publication-ready draft.
Workflow E: Blind A/B Benchmark Duel (Style Discovery)
Goal: Objectively determine which frontier model writes best for your personal voice without brand bias.
You:
"Benchmark the best models for writing an executive pitch email to CTOs."
Claude:
Calls
benchmark_create_task(title: "CTO Pitch", category: "Emails", prompt: "...").Calls
benchmark_generate_outputs(taskId)to query competing models in parallel via OpenRouter.Calls
benchmark_start_duel(taskId)and displays anonymous options:Response A:
[Anonymous output]Response B:
[Anonymous output]Which response is more compelling, A or B?
You vote: "Response A is punchier and highlights developer productivity much better."
Claude calls
benchmark_submit_vote(battleId, choice: "A").When satisfied, you ask: "Reveal results!"
Claude calls
benchmark_get_results(taskId, reveal: true):Response A was Claude 3.5 Sonnet (Score: 1240, Latency: 780ms)
Response B was GPT-4o (Score: 1195, Latency: 910ms)
Claude calls
benchmark_get_leaderboardandbenchmark_analyze_preferencesto update your personal ranking!
9. Mathematical Ranking Details
Bradley-Terry Maximum Likelihood Estimation (MLE)
Pairwise choice probabilities follow: $$P(i \succ j) = \frac{\pi_i}{\pi_i + \pi_j} = \frac{e^{\beta_i}}{e^{\beta_i} + e^{\beta_j}}$$
The model latent parameters $\pi$ are solved using the iterative Minorization-Maximization (MM / Hunter 2004) algorithm with Laplace prior regularization ($\alpha = 1.0$) for numerical stability with disconnected matchup graphs: $$\pi_i^{(t+1)} = \frac{W_i + \alpha}{\sum_{j \ne i} \frac{N_{ij}}{\pi_i^{(t)} + \pi_j^{(t)}} + \alpha}$$
Uncertainty Quantification: Standard errors are derived from the observed Fisher Information matrix:
$< 5$ battles: Preliminary (High Uncertainty)
$5β15$ battles: Emerging (Moderate Uncertainty)
$> 15$ battles: Established (High Confidence)
10. Testing
The project includes an extensive test suite verifying mathematical correctness, zero-bias coin toss distributions, strict anonymity boundaries, and end-to-end tournament simulations:
# Run all unit and integration tests
npm test
# Run tests in watch mode
npm run test:watch11. License
MIT Β© Yadnyesh Borole.
Available Tools
11 toolsbenchmark_analyze_preferencesB
Analyze empirical user preference patterns (conciseness, structure, tone) from battle voting history.
Inputs: category, min_votes.
benchmark_compare_modelsB
Compare two AI models head-to-head using accumulated pairwise benchmark battle history.
Inputs: model_a_id (required), model_b_id (required), category.
benchmark_create_taskA
Create a new blind AI writing benchmark task with a category, prompt, and optional evaluation criteria.
Inputs: title (required), prompt (required), category (required), user_id, +2 more.
benchmark_generate_outputsA
Generate writing outputs for a benchmark task across competing models via OpenRouter. Models remain strictly anonymous.
Inputs: task_id (required), model_ids, temperature.
benchmark_get_leaderboardA
Get personal or global model rankings calculated with Bradley-Terry Maximum Likelihood Estimation and Elo.
Inputs: scope, metric, category, min_battles.
benchmark_get_model_statsA
Get comprehensive performance metrics, win rates, and ranking score for a specific model.
Inputs: model_id (required).
benchmark_get_resultsB
Retrieve benchmark results for a task. When reveal is true, model identities are unmasked.
Inputs: task_id (required), reveal.
benchmark_list_modelsB
List all AI writing models available in the benchmark registry with pricing information.
Inputs: category, enabled_only.
benchmark_start_duelB
Start a randomized, blind A/B battle between two outputs for evaluation. Model identities remain masked.
Inputs: task_id (required), output_a_id, output_b_id.
benchmark_submit_voteA
Submit a blind user preference vote ('A', 'B', or 'tie') with optional reasoning and dimension ratings.
Inputs: choice (required), battle_id (required), reason, dimension_scores.
writer_generateA
PRIMARY WRITING TOOL. Use this tool whenever the user asks to write, draft, or compose content (emails, articles, proposals, essays, sales copy, social posts). Instead of generating long-form drafts with Claude output tokens, first outline the strategy and key arguments, then call this tool to delegate the draft generation to OpenRouter models (DeepSeek V3, Llama 3.3, GPT-4o, etc.). Automatically selects the user's #1 ranked model from their personal leaderboard or cost-effective DeepSeek V3.
Inputs: prompt (required), category, model_id, max_tokens, +2 more.
Tool Schema Changelog
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelogβ¦
TDQS
Scored across 11 tools
Most benchmark tools have clearly distinct resource-action boundaries, but benchmark_compare_models, benchmark_get_model_stats, and benchmark_get_leaderboard all describe performance summaries and could be confused by an agent. benchmark_generate_outputs and writer_generate are somewhat related, though writer_generate is explicitly marked as the primary writing tool.
Ten tools consistently follow a benchmark_<verb>_<noun> naming pattern, which is highly predictable. writer_generate breaks the pattern by using noun_verb form and dropping the benchmark_ prefix, making the set slightly inconsistent.
Eleven tools is well-scoped for a blind benchmark workflow plus an integrated writing generation tool. Each tool serves a recognizable step or query in the system, and the count is neither bloated nor thin.
The core benchmark lifecycle is covered: create task, generate outputs, start duel, submit vote, get results, and retrieve leaderboards. Obvious gaps include no list/update/delete for tasks and no way to enumerate active duels, but these are workable minor omissions.
Maintenance
Related MCP Connectors
AI routing, memory, guardrails, and governance. Routes across Claude, GPT, Gemini.
AI model routing on your own vendor keys: pick the best model per prompt, or route and run it.
BYOK AI router & FinOps control plane: audit token waste, check tool safety, and rank models.
Checks if a cheaper model agrees with your GPT, Claude or Gemini outputs, and what routing saves.
Related MCP Servers
- AlicenseBqualityDmaintenanceRoute prompts intelligently across Claude, Gemini, and GPT-4o, automatically picking the best model for every task while minimizing token cost.523 npmMIT
- FlicenseNot gradedqualityCmaintenanceEnables adversarial multi-model AI review via OpenRouter, allowing Claude to submit prompts to multiple models and compare responses through custom connector tools.-
- AlicenseNot gradedqualityBmaintenanceEnables AI agents to route chat completion requests to task-appropriate models via LiteLLM/OpenRouter, track usage and budgets, and run side-by-side eval comparisons of models based on cost, latency, and response quality.116 npmMIT
- AlicenseBqualityBmaintenanceEnables Claude Code and Claude Desktop to delegate bounded, disposable tasks to local or cheap cloud models, with model scoring, guardrails, and cost-saved tracking.53MIT