Click on "Deploy Server".
Wait a few minutes for the server to deploy. Once ready, it will show a "Started" state.
In the chat, type @ followed by the MCP server name and your instructions, e.g., "@com.woodstitch/tts-human-eval Price a blinded pairwise study comparing my two TTS clips, 5 ratings per pair."
That's it! The server will respond to your query, and you can continue using it as needed.
Here is a step-by-step guide with screenshots.
An MCP server for the Woodstitch API: paid human listeners judge your text-to-speech audio in blinded studies, and your agent gets a number back with a confidence interval.
Two tasks, and only two: pairwise preference (a listener hears two clips of the same text and picks one, forced choice) and human-vs-AI detection (one clip, human or AI, confidence 1–4). Not offered: MOS, CMOS, MUSHRA or any 1–5 scale.
Get an API key at https://woodstitch.com/app/api-keys, then:
Claude Code
claude mcp add woodstitch -e WOODSTITCH_API_KEY=ws_live_... -- uvx woodstitch-mcpClaude Desktop / Cursor / any MCP client (mcpServers entry):
{
"woodstitch": {
"command": "uvx",
"args": ["woodstitch-mcp"],
"env": {"WOODSTITCH_API_KEY": "ws_live_..."}
}
}pipx run woodstitch-mcp or pip install woodstitch-mcp && woodstitch-mcp work too.
Requires Python 3.10+.
Claude Code users: the Woodstitch plugin bundles this server
with a skill that knows when and how to run a listening test:
/plugin marketplace add woodstitch-llc/claude-plugin then /plugin install tts-human-eval@woodstitch.
Related MCP server: loop-tts
Tool | What it does | Money |
| What is measured and what is not (no key needed) | – |
| Rate card and worked examples (no key needed) | – |
| Balance and spend | – |
| Upload local audio, get file ids | – |
| Create and price a study; | charges nothing |
| Start the study | spends money |
| Progress and provisional score | – |
| Final numbers | – |
| Every judgement as rows (json or csv, optionally to a file) | – |
| Stop early, pay only for work done | – |
| housekeeping | – |
woodstitch_start_eval requires user_confirmed_amount_usd, the quoted total the user agreed
to, and sends it to the API as the hard ceiling max_total_usd. If the quote is higher than
what the user confirmed, the API refuses (over_max) and nothing starts. An agent cannot start
a study the user has not seen the price of.
Every POST carries an Idempotency-Key, so a retried call never creates or starts a study twice.
The server's instructions tell the model that interim numbers are provisional and that stopping early on a threshold is optional stopping.
woodstitch_upload_files(["a/001.wav", "b/001.wav", ...]) or use public URLs.
woodstitch_create_eval(pairs=[{"id":"001","a":"file_…","b":"file_…","passage":"001","system_a":"base","system_b":"candidate"}, …], ratings_per_pair=5) → quote, nothing charged.
Show the user the total. On yes: woodstitch_start_eval(eval_id, user_confirmed_amount_usd=<total>).
woodstitch_get_eval(eval_id) now and then; first judgements usually land within minutes.
woodstitch_get_results(eval_id) when status is completed; woodstitch_export for the rows.
Method, listener pay and controls: https://woodstitch.com/research/methodology. Machine-readable API spec: https://woodstitch.com/openapi.json.
uv venv && . .venv/bin/activate && uv pip install -e .
WOODSTITCH_API_KEY=ws_live_... woodstitch-mcp # stdio serverMIT © Woodstitch LLC
Account balance and spend for the configured API key.
Inputs: no named parameters.
Stop an evaluation early. Charges only for work already handed to listeners; anyone mid-task finishes and is paid. Check max_additional_charge_if_canceled_now_usd via woodstitch_get_eval first if the user wants that number. Requires confirm=true.
Inputs: eval_id (required), confirm, idempotency_key.
Create and PRICE an evaluation. Charges nothing. Resolves every clip (file id or public URL), validates it, measures duration and returns a quote with per-pair prices, warnings and the compiled design. Set dry_run=true to price without storing anything (no eval_id). Preference: pass pairs as [{id, a, b, system_a?, system_b?, passage?, speaker?}] where a/b are file ids or URLs rendering the SAME text; sides are shown to listeners in random order. Detection: pass clips as [{id, audio, truth: 'human'|'ai', source, speaker, passage, duration_bucket?}] and include real human recordings of the same passages as the baseline. per_participant = {max_judgements, unique_by: [...], max_per: {...}} limits what one listener hears; for a multi-system comparison use unique_by ['passage'] and max_per {'system': N}. ALWAYS show the returned quote total to the user before starting.
Inputs: name, clips, pairs, dry_run, +7 more.
Delete one upload from your account by file id. Studies that already used the file keep their copy.
Inputs: file_id (required).
Every judgement as a row (pseudonymous rater, pair/clip id, source, speaker, passage, answer, confidence, time spent). format 'json' returns the rows; 'csv' returns CSV text. Pass save_to (a local path) to write the export to disk instead of returning it, which is better for large studies.
Inputs: eval_id (required), format, save_to.
Progress and live score for an evaluation: status, judgements done/target, pairs with data, provisional metrics, dashboard_url, and max_additional_charge_if_canceled_now_usd. Cheap to poll. Humans take minutes to days; do not sit in a tight loop. Interim numbers are provisional.
Inputs: eval_id (required).
Final numbers once the evaluation is completed (or canceled, then incomplete=true). Preference: human_pref/b_win_rate is the share of preference for your b side with a 95% CI; 0.5 = indistinguishable. Detection: human_label_rate per source with two intervals: tested_stimuli (these files) and model (generalises over voices and passages; 3-4x wider; quote THIS one for any claim about a model). Returns not_ready while still running.
Inputs: eval_id (required).
Your last 50 evaluations with status and progress.
Inputs: no named parameters.
List your uploaded files, newest first.
Inputs: no named parameters.
Rate card: USD per hour of audio a listener hears, priced per judgement from the measured duration of your clips, with worked examples and limits. No API key needed.
Inputs: no named parameters.
START an evaluation. THIS SPENDS REAL MONEY and cannot be undone once listeners begin. Only call it after the user has seen the quote total returned by woodstitch_create_eval and explicitly agreed. Pass that total as user_confirmed_amount_usd: it is sent to the API as the hard spending ceiling (max_total_usd), so if the current quote is higher than what the user confirmed the API refuses with over_max and nothing starts. Billing is metered from the first judgement; a study pauses rather than overspends if the balance runs out.
Inputs: eval_id (required), user_confirmed_amount_usd (required), quote_id, idempotency_key.
What Woodstitch measures: the two templates (tts_preference.v1: two clips, forced choice; tts_detect.v1: one clip, human or AI + confidence), their inputs and metrics, and what is NOT offered (MOS, CMOS, MUSHRA, any 1-5 scale). No API key needed.
Inputs: no named parameters.
Upload audio files from the local disk (wav, mp3, flac, ogg, m4a; 0.5-120 s each, 50 MB per file, up to 100 files per call). Returns file ids, measured durations and formats, in the order sent. Use an id anywhere woodstitch_create_eval takes an audio reference. Re-uploading the same bytes is harmless (content-addressed).
Inputs: paths (required).
Recent tool additions, removals, and schema changes observed during successful MCP inspections.
Loading tool schema changelog…
Scored across 13 tools
Each tool targets a distinct resource and action: informational lookups (templates, pricing), file management (upload/list/delete), and a clearly separated eval lifecycle (create/start/get/results/export/cancel). The potentially confusable pairs are explicitly differentiated — create_eval only prices while start_eval spends money, and get_eval returns live progress while get_results returns final metrics.
All names share the woodstitch_ prefix and are snake_case, with verb_noun patterns for the operations (list_evals, upload_files, create_eval, cancel_eval). The three informational tools (templates, pricing, account) are bare nouns rather than verb_noun, which is a minor but sensible deviation since they take no action.
Thirteen tools is well within the sweet spot and every tool maps to a real step in the workflow: learn the product, upload clips, price, start, monitor, collect results, export, cancel. Nothing looks redundant or padded.
The full lifecycle is covered — discovery, file CRUD, quoting, starting, polling, results, export, cancellation, and account balance — with no obvious dead ends. Minor gaps remain, such as no way to delete an eval record or re-inspect a stored eval's compiled design after creation (only re-pricing via dry_run).
Test the voice agents you run: scored transcripts, pass/fail verdicts, latency and WER metrics.
Run user and customer interviews with real people from your AI agent, by MCP or REST API. Vetted, fraud-screened participants from a 4M+ consumer panel or your own list. AI moderation that ladders into why. Voice, video or chat in 80+ languages. Findings come back cited to quotes.
Human judgment for AI agents: discover capabilities, get quotes, and track paid human tasks.
Run data annotation and evaluation tasks with vetted domain experts: create, publish, get results.
