Skip to main content
Glama
com.woodstitch/tts-human-eval

com.woodstitch/tts-human-eval

Official

woodstitch-mcp

An MCP server for the Woodstitch API: paid human listeners judge your text-to-speech audio in blinded studies, and your agent gets a number back with a confidence interval.

Two tasks, and only two: pairwise preference (a listener hears two clips of the same text and picks one, forced choice) and human-vs-AI detection (one clip, human or AI, confidence 1–4). Not offered: MOS, CMOS, MUSHRA or any 1–5 scale.

Install

Get an API key at https://woodstitch.com/app/api-keys, then:

Claude Code

claude mcp add woodstitch -e WOODSTITCH_API_KEY=ws_live_... -- uvx woodstitch-mcp

Claude Desktop / Cursor / any MCP client (mcpServers entry):

{
  "woodstitch": {
    "command": "uvx",
    "args": ["woodstitch-mcp"],
    "env": {"WOODSTITCH_API_KEY": "ws_live_..."}
  }
}

pipx run woodstitch-mcp or pip install woodstitch-mcp && woodstitch-mcp work too. Requires Python 3.10+.

Claude Code users: the Woodstitch plugin bundles this server with a skill that knows when and how to run a listening test: /plugin marketplace add woodstitch-llc/claude-plugin then /plugin install tts-human-eval@woodstitch.

Related MCP server: loop-tts

Tools

Tool

What it does

Money

woodstitch_templates

What is measured and what is not (no key needed)

–

woodstitch_pricing

Rate card and worked examples (no key needed)

–

woodstitch_account

Balance and spend

–

woodstitch_upload_files

Upload local audio, get file ids

–

woodstitch_create_eval

Create and price a study; dry_run to price only

charges nothing

woodstitch_start_eval

Start the study

spends money

woodstitch_get_eval

Progress and provisional score

–

woodstitch_get_results

Final numbers

–

woodstitch_export

Every judgement as rows (json or csv, optionally to a file)

–

woodstitch_cancel_eval

Stop early, pay only for work done

–

woodstitch_list_evals, woodstitch_list_files, woodstitch_delete_file

housekeeping

–

Safeguards built in

  • woodstitch_start_eval requires user_confirmed_amount_usd, the quoted total the user agreed to, and sends it to the API as the hard ceiling max_total_usd. If the quote is higher than what the user confirmed, the API refuses (over_max) and nothing starts. An agent cannot start a study the user has not seen the price of.

  • Every POST carries an Idempotency-Key, so a retried call never creates or starts a study twice.

  • The server's instructions tell the model that interim numbers are provisional and that stopping early on a threshold is optional stopping.

Typical session

  1. woodstitch_upload_files(["a/001.wav", "b/001.wav", ...]) or use public URLs.

  2. woodstitch_create_eval(pairs=[{"id":"001","a":"file_…","b":"file_…","passage":"001","system_a":"base","system_b":"candidate"}, …], ratings_per_pair=5) → quote, nothing charged.

  3. Show the user the total. On yes: woodstitch_start_eval(eval_id, user_confirmed_amount_usd=<total>).

  4. woodstitch_get_eval(eval_id) now and then; first judgements usually land within minutes.

  5. woodstitch_get_results(eval_id) when status is completed; woodstitch_export for the rows.

Method, listener pay and controls: https://woodstitch.com/research/methodology. Machine-readable API spec: https://woodstitch.com/openapi.json.

Development

uv venv && . .venv/bin/activate && uv pip install -e .
WOODSTITCH_API_KEY=ws_live_... woodstitch-mcp        # stdio server

MIT © Woodstitch LLC

Available Tools

13 tools
woodstitch_accountA
Read-only

Account balance and spend for the configured API key.

NameRequiredDescriptionDefault

No parameters

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, openWorldHint=true, idempotentHint=false and destructiveHint=false, so the safety profile is covered. The description adds one useful behavioral fact — results are tied to the configured API key rather than a passed-in account — but says nothing about freshness, rate limits, or what the payload contains.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single short sentence with no filler, front-loading the returned data (balance and spend) before the scoping clause. Nothing is redundant with the name.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-param read tool this covers what it returns at a high level, but there is no output schema, so the description is the only place to state the response shape (currency, units, period for "spend", field names) and it does not. Adequate to select the tool, thin for interpreting results.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so there is no parameter meaning to convey; baseline for a parameterless schema is 4. The phrase "for the configured API key" usefully confirms that account scope comes from credentials, not input.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a concrete resource pair (balance and spend) scoped to "the configured API key," so an agent knows this is a retrieval of its own account state rather than the workspace/object tools. No explicit verb is used, and it does not name a sibling it differs from, but the purpose is unambiguous against siblings like woodstitch_pricing or woodstitch_list_files.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: an agent can infer you call this to check billing/credit status, and the "configured API key" phrasing implies no account-selection step is needed. There is no explicit when-to-use or when-not-to-use statement, and no reference to alternatives such as woodstitch_pricing for rate/cost information.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

woodstitch_cancel_evalA
Destructive

Stop an evaluation early. Charges only for work already handed to listeners; anyone mid-task finishes and is paid. Check max_additional_charge_if_canceled_now_usd via woodstitch_get_eval first if the user wants that number. Requires confirm=true.

NameRequiredDescriptionDefault
confirmNo
eval_idYes
idempotency_keyNo

TDQS

A4.1/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare destructiveHint=true and readOnlyHint=false, but the description goes further by explaining the billing model: charges apply only to work already handed to listeners, and in-flight tasks finish and are paid. That is exactly the cost/irreversibility context an agent needs before a destructive call. It omits what happens if confirm is false and how idempotency_key behaves, keeping it short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short sentences with the core action front-loaded, followed by cost implications and the prerequisite. The get_eval pointer is slightly lumped together with pricing vocabulary, but no sentence is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a destructive, billing-relevant mutation with no output schema and zero schema description coverage, the description covers the cost outcome and the confirmation gate, which is the bulk of what matters. The remaining gap is the idempotency_key semantics and the failure behavior when confirm is not set.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must carry parameter meaning itself. It explains the confirm=true requirement, which is the non-obvious gate on this call, and eval_id is self-evident from the name, but idempotency_key is never mentioned in either schema or description, leaving one of three parameters undocumented.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Stop an evaluation early') that is immediately distinguishable from the sibling lifecycle tools start_eval, get_eval, and list_evals. An agent knows exactly what action this performs without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a clear condition for consulting woodstitch_get_eval first ('if the user wants that number') and states the confirm=true prerequisite, which routes the agent correctly before invoking. It stops short of naming explicit when-not-to-cancel conditions or alternatives beyond the pricing check, so it is strong but not exhaustive.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

woodstitch_create_evalA
Idempotent

Create and PRICE an evaluation. Charges nothing. Resolves every clip (file id or public URL), validates it, measures duration and returns a quote with per-pair prices, warnings and the compiled design. Set dry_run=true to price without storing anything (no eval_id). Preference: pass pairs as [{id, a, b, system_a?, system_b?, passage?, speaker?}] where a/b are file ids or URLs rendering the SAME text; sides are shown to listeners in random order. Detection: pass clips as [{id, audio, truth: 'human'|'ai', source, speaker, passage, duration_bucket?}] and include real human recordings of the same passages as the baseline. per_participant = {max_judgements, unique_by: [...], max_per: {...}} limits what one listener hears; for a multi-system comparison use unique_by ['passage'] and max_per {'system': N}. ALWAYS show the returned quote total to the user before starting.

NameRequiredDescriptionDefault
nameNo
clipsNo
pairsNo
dry_runNo
questionNo
templateNotts_preference.v1
allocationNo
idempotency_keyNo
per_participantNo
ratings_per_clipNo
ratings_per_pairNo

TDQS

A4.1/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Adds substantial behavior beyond the annotations: no charges are incurred, every clip is resolved (file id or public URL), validated, and duration-measured, and the response includes per-pair prices, warnings, and the compiled design. It also discloses that dry_run persists nothing and returns no eval_id, and that a/b sides are shown in random order — none of which is inferable from readOnlyHint=false / destructiveHint=false / idempotentHint=true.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the verb, the resource, and the zero-cost fact, then organized by task (pairs → detection → allocation → warning). It is dense and somewhat run-on, with multiple ideas packed into single sentences, but nearly every clause carries operational information rather than filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description takes on the return-value burden and does so for the pricing payload (quote total, per-pair prices, warnings, compiled design) and the dry_run/no-eval_id distinction. It stops short of explaining what a non-dry-run call yields for follow-up (eval_id, next step being woodstitch_start_eval), which matters for an 11-parameter creation tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% and the schema types `clips`, `pairs`, `allocation`, and `per_participant` as untyped objects, so the description carries real weight — it supplies element shapes for `pairs` and `clips`, the dry_run semantics, and the per_participant keys. However, 7 of 11 parameters (name, question, template, allocation, idempotency_key, ratings_per_clip, ratings_per_pair) get no mention at all, leaving a significant compensation gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource — 'Create and PRICE an evaluation' — with an unusual dual nature (create + quote) made explicit up front, plus 'Charges nothing' to disambiguate cost. It does not name the adjacent siblings (woodstitch_pricing, woodstitch_start_eval), so an agent must still infer the boundary between pricing-only, create, and start.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives concrete when-to-use branches: dry_run=true to price without storing, `pairs` for preference tests, `clips` for detection tests, and a specific per_participant recipe for multi-system comparisons. It even imposes an operational rule ('ALWAYS show the returned quote total to the user before starting'). What is missing is explicit routing against woodstitch_pricing and woodstitch_start_eval.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

woodstitch_delete_fileB
Destructive

Delete one upload from your account by file id. Studies that already used the file keep their copy.

NameRequiredDescriptionDefault
file_idYes

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare destructiveHint=true, readOnlyHint=false, and idempotentHint=false, so the safety profile is covered by structured data. The description does add real value beyond that by explaining that studies which already used the file retain their copy, i.e. deletion is not cascading. It omits permission requirements and behavior when the id does not exist.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences with the operation and its scope front-loaded, followed by one non-obvious caveat. Nothing is padded, though the second sentence could be tied more explicitly to the deletion action.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter destructive tool with no output schema, the annotations supply the safety signals and the description adds the non-cascading guarantee. Only the provenance of file_id and not-found behavior are left unaddressed, which are minor gaps.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

With one parameter and 0% schema description coverage, the description carries the full burden, but it only restates 'file id' without format, example, or where to obtain a valid id (presumably woodstitch_list_files). This is the weakest area of an otherwise competent definition.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Delete one upload') plus the scoping unit ('one', 'by file id'), which clearly separates it from woodstitch_upload_files and woodstitch_list_files. An agent can identify the operation without opening the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description never says when to reach for this tool versus alternatives such as woodstitch_list_files (to find an id) or whether deletion is ever preferable to leaving a file in place. The second sentence explains a side effect of deleting, not when to delete.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

woodstitch_exportA
Read-only

Every judgement as a row (pseudonymous rater, pair/clip id, source, speaker, passage, answer, confidence, time spent). format 'json' returns the rows; 'csv' returns CSV text. Pass save_to (a local path) to write the export to disk instead of returning it, which is better for large studies.

NameRequiredDescriptionDefault
formatNojson
eval_idYes
save_toNo

TDQS

A3.5/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and destructiveHint=false, so the safety profile is known. The description adds real value by explaining the format switch and that save_to writes a file to a local path instead of returning data. It omits how rows are ordered, whether the export is complete or truncated, and what happens if eval_id is unknown.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences, front-loaded with the payload shape, then format semantics, then the disk-write option. The opening fragment 'Every judgement as a row' is slightly clipped but dense with useful detail; nothing is redundant.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description correctly spends its words enumerating the returned columns and both return modes, which is what an agent needs to interpret the result. The remaining gap is the required eval_id and any indication of export size or failure behavior.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description carries the whole burden. It does define format's two values and save_to's behavior well, but eval_id, the only required parameter, is never explained beyond its name.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states the resource precisely: one row per judgement, with the exact column list (rater, pair/clip id, source, speaker, passage, answer, confidence, time spent). That is far more specific than the bare tool name. It does not distinguish itself from the sibling woodstitch_get_results, which an agent would plausibly consider for the same goal.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives one genuine conditional: pass save_to for large studies rather than returning the payload. However, it never says when to use this export versus woodstitch_get_results or woodstitch_list_evals, and it states no prerequisites for a valid eval_id.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

woodstitch_get_evalA
Read-only

Progress and live score for an evaluation: status, judgements done/target, pairs with data, provisional metrics, dashboard_url, and max_additional_charge_if_canceled_now_usd. Cheap to poll. Humans take minutes to days; do not sit in a tight loop. Interim numbers are provisional.

NameRequiredDescriptionDefault
eval_idYes

TDQS

A4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already establish that this is a non-destructive read operation, and the description adds useful behavioral context: polling is cheap, completion can take minutes to days, tight-loop polling is discouraged, and interim metrics are provisional. It also discloses key returned fields such as dashboard_url and max_additional_charge_if_canceled_now_usd, which is valuable because no output schema exists.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is compact and front-loaded: it first names the return content, then gives polling behavior and timing expectations. Every sentence adds useful information without repetition.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter read tool with no output schema, the description is largely complete: it lists the returned fields, discloses polling cost and timing, and warns that interim numbers are provisional. The main remaining gap is that it does not explain how eval_id is obtained or when to prefer this over final-results/list-evals siblings.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

There is one required parameter, eval_id, with 0% schema description coverage. The description implies it identifies the evaluation being monitored, but it does not explain how to obtain eval_id or add format/source guidance beyond what the parameter name and type already convey.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description clearly states the tool returns progress and live score for an evaluation, including status, judgements done/target, pairs with data, provisional metrics, dashboard_url, and cancellation charge. It is specific about the resource and return content, though it does not explicitly distinguish itself from siblings such as woodstitch_get_results or woodstitch_list_evals.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description gives clear polling guidance: it is cheap to poll, human evaluations take minutes to days, and the agent should not sit in a tight loop. It also notes interim numbers are provisional, which helps set expectations. However, it does not explicitly say when to use this versus final-results or list-evals siblings.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

woodstitch_get_resultsA
Read-only

Final numbers once the evaluation is completed (or canceled, then incomplete=true). Preference: human_pref/b_win_rate is the share of preference for your b side with a 95% CI; 0.5 = indistinguishable. Detection: human_label_rate per source with two intervals: tested_stimuli (these files) and model (generalises over voices and passages; 3-4x wider; quote THIS one for any claim about a model). Returns not_ready while still running.

NameRequiredDescriptionDefault
eval_idYes

TDQS

A4.1/5.0
Behavior5/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations, the description explains readiness behavior, the incomplete flag for canceled evaluations, and how to interpret preference and detection intervals. It even warns which interval to quote for model-level claims. This is substantial behavioral and interpretive context for a read-only results endpoint.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The description is dense and front-loads the key availability condition before explaining metric semantics. It is appropriately compact for the amount of interpretation it conveys, though the run-on structure makes it harder to scan.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the burden of explaining return behavior and does so for readiness, incomplete results, and key result metrics. It is not fully complete because it omits parameter sourcing and does not enumerate all possible result fields.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema has 0% description coverage, and the description never explains eval_id, where to obtain it, or how it relates to list/create/start siblings. With only one parameter this is somewhat inferable from the name, but the description does not compensate for the missing schema documentation.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific resource and outcome: final results once an evaluation is completed or canceled. It also distinguishes the result retrieval moment from the running state by noting not_ready while still running. The metric explanations further confirm this is a results-reading tool rather than a status/listing tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives clear timing guidance: call after completion or cancellation, and expect not_ready while the evaluation is running. It does not explicitly name an alternative sibling such as get_eval for checking status, but the availability condition is clear.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

woodstitch_list_evalsB
Read-only

Your last 50 evaluations with status and progress.

NameRequiredDescriptionDefault

No parameters

TDQS

B3.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and destructiveHint=false, so safety is covered. The description adds genuinely useful behavioral detail beyond that: a fixed window of 50 records and the fields returned (status, progress). It omits ordering (newest first?), whether there is any way to page beyond 50, and whether results are scoped to the caller's account.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single short sentence with no padding, and the key constraint (50) and return fields are front-loaded. It is slightly under-specified rather than verbose, and lacks an explicit action verb.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema and no parameters, the description carries the whole burden of explaining what comes back, and it only names two fields. For a zero-arg list tool an agent still needs to know sort order, the fixed 50-item ceiling with no pagination path, and scope of 'your' evaluations.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so the baseline is 4; there is nothing for the description to disambiguate. The mention of 'last 50' at least signals that the selection is fixed rather than parameterized.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The fragment names the resource (evaluations) and a result shape (status and progress) plus a hard limit of 50, so an agent can infer a listing operation. But there is no explicit verb and nothing distinguishes it from siblings like woodstitch_get_eval or woodstitch_get_results, which also surface evaluation state.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

No when-to-use guidance at all: it never says to call this for an overview versus woodstitch_get_eval for a single run or woodstitch_get_results for outputs. The 'last 50' framing implies a recent-activity use case, but only by inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

woodstitch_list_filesA
Read-only

List your uploaded files, newest first.

NameRequiredDescriptionDefault

No parameters

TDQS

A3.6/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, destructiveHint=false and openWorldHint=true, so the safety profile is covered. The description adds genuinely new behavioral context in 'newest first' (result ordering), but says nothing about pagination, result caps, or whether all files or only a page are returned.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single short sentence with zero filler, and the resource plus ordering are front-loaded. Every clause carries information an agent can act on.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter read-only tool this is close to adequate, but with no output schema the description could have briefly characterized the returned list (file identifiers, metadata, pagination) to help an agent plan follow-up calls such as get_eval or delete_file.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters and the schema is an empty object, so there is no parameter semantics to explain. The baseline of 4 applies; nothing in the description conflicts with the empty schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('List your uploaded files') plus the sort order ('newest first'), which is enough to distinguish it from siblings like woodstitch_list_evals and woodstitch_templates. It does not explicitly name those siblings, so it stops short of the top score.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: the word 'list' makes the intent obvious, and there are no parameters to get wrong. However, the description never states when to prefer this over siblings (e.g., list_evals) or points to upload_files/delete_file for follow-up actions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

woodstitch_pricingA
Read-only

Rate card: USD per hour of audio a listener hears, priced per judgement from the measured duration of your clips, with worked examples and limits. No API key needed.

NameRequiredDescriptionDefault

No parameters

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations cover the safety profile (readOnly, non-destructive), so the description's job is to add context, which it does: it explains how price is computed (per judgement, from measured duration), that the response contains worked examples and limits, and that no API key is required. It does not mention freshness of the rate card or whether prices vary by plan.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence led by "Rate card", with every clause carrying information (currency, unit, pricing basis, contents, auth). The phrasing around "priced per judgement from the measured duration" is dense and takes a second read.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema and no parameters, the description carries the burden of describing the return content, and it does so adequately — rate per hour, worked examples, and limits. A short note on when the rates apply or how they are kept current would close the remaining gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so the baseline is 4. Nothing in the description is needed to explain inputs and none is wasted on them.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

"Rate card: USD per hour of audio" names the resource (pricing) and the unit of pricing (USD per hour) with a specific basis (measured clip duration, judged per judgement). It is clearly not an action tool like the eval-management siblings, but it never explicitly contrasts itself with them.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied — an agent infers it should call this when it needs cost information. There is no explicit "use this when you need to estimate cost before creating an eval" statement or reference to any alternative source of pricing information.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

woodstitch_start_evalA

START an evaluation. THIS SPENDS REAL MONEY and cannot be undone once listeners begin. Only call it after the user has seen the quote total returned by woodstitch_create_eval and explicitly agreed. Pass that total as user_confirmed_amount_usd: it is sent to the API as the hard spending ceiling (max_total_usd), so if the current quote is higher than what the user confirmed the API refuses with over_max and nothing starts. Billing is metered from the first judgement; a study pauses rather than overspends if the balance runs out.

NameRequiredDescriptionDefault
eval_idYes
quote_idNo
idempotency_keyNo
user_confirmed_amount_usdYes

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

The description goes well beyond the annotations: it discloses real monetary spend, irreversibility once listeners begin, the hard ceiling mechanism via max_total_usd, the over_max refusal path, metered billing from the first judgement, and the pause-rather-than-overspend behavior. The only gap is that idempotentHint=false is never addressed, so the agent gets no guidance on retry safety for a costly call.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

The action and the money warning are front-loaded in the first sentence, followed by precondition, parameter semantics, and billing model in a logical order. Every sentence carries distinct operational information with no filler.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a costly, open-world mutation with no output schema, the description covers the critical safety and sequencing context an agent needs. It stops short of explaining what the call returns on success or the role of quote_id and idempotency_key, which leaves some operational detail to inference.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must carry parameter meaning, and it does so thoroughly for user_confirmed_amount_usd — explaining it is transmitted as max_total_usd and acts as the spending ceiling. However, eval_id, quote_id, and idempotency_key receive no explanation at all, leaving half the parameters undocumented.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

"START an evaluation" is a specific verb plus resource, and the surrounding text makes clear this is the execution step that follows quoting, not a listing or creation tool. An agent can distinguish it from woodstitch_create_eval and woodstitch_cancel_eval without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicit when-to-use is given: only after the user has seen the total from woodstitch_create_eval and explicitly agreed. It also names the consequence of skipping that step (API refuses with over_max), so the sequencing requirement is unambiguous.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

woodstitch_templatesA
Read-only

What Woodstitch measures: the two templates (tts_preference.v1: two clips, forced choice; tts_detect.v1: one clip, human or AI + confidence), their inputs and metrics, and what is NOT offered (MOS, CMOS, MUSHRA, any 1-5 scale). No API key needed.

NameRequiredDescriptionDefault

No parameters

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true and destructiveHint=false, so the description's job is to add context — and it does, stating 'No API key needed' (an auth trait absent from the annotations) and enumerating what the payload covers and explicitly excludes. It stops short of describing the exact response shape, but that is a minor gap for a static reference call.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence stating the subject, the two template names, and the exclusions, with no filler. The nested parentheses and mixed colons/semicolons make it slightly dense to parse quickly, which keeps it from a 5.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no parameters, no output schema, and annotations already covering safety and openness, the description supplies everything an agent needs: which templates exist, what each measures, and what is deliberately not offered. Nothing required for correct invocation is absent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so there is nothing for the description to disambiguate; the schema is empty and complete. Baseline 4 applies, with no parameter-level meaning needed or missing.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names the concrete resource it covers — the two Woodstitch templates (tts_preference.v1 with two-clip forced choice, tts_detect.v1 with one clip plus human/AI confidence) and their inputs and metrics. It also draws a sharp negative boundary (no MOS, CMOS, MUSHRA, no 1-5 scale), which no sibling would provide. The verb is implicit ('What Woodstitch measures') rather than an explicit 'returns documentation for', keeping it short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied: an agent would consult this before choosing a template for woodstitch_create_eval or interpreting woodstitch_get_results. However, it never says 'call this before creating an eval' and never names a sibling such as woodstitch_pricing or woodstitch_create_eval as the alternative for a different need.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

woodstitch_upload_filesA
Idempotent

Upload audio files from the local disk (wav, mp3, flac, ogg, m4a; 0.5-120 s each, 50 MB per file, up to 100 files per call). Returns file ids, measured durations and formats, in the order sent. Use an id anywhere woodstitch_create_eval takes an audio reference. Re-uploading the same bytes is harmless (content-addressed).

NameRequiredDescriptionDefault
pathsYes

TDQS

A4.5/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Beyond the annotations it discloses validation constraints (duration bounds, 50 MB per file, 100 files per call) and the shape and ordering of results, which the agent cannot infer from any structured field. The 're-uploading the same bytes is harmless (content-addressed)' line adds the mechanism behind idempotentHint=true rather than merely restating it. Safe-write/idempotency/closed-world semantics are already covered by annotations, so this is additive context, not full disclosure of error behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Four dense sentences, front-loaded with the action and accepted formats, then constraints, then return shape, then downstream use. Every clause carries information an agent needs; nothing is redundant padding.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description still specifies what comes back (file ids, measured durations, formats, in sent order), which is the critical unknown for a tool whose ids feed into woodstitch_create_eval. Formats, size/duration/count limits, and idempotency are all present, so an agent can call this correctly on the first attempt.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The schema provides a bare array of strings with 0% description coverage, so the description carries the load: it explains these are local-disk audio paths, lists accepted extensions, and constrains the batch to 100 items and 50 MB each. It still does not say whether paths must be absolute, whether directories/globs are accepted, or what happens on a partial failure.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Upload audio files from the local disk') and immediately bounds it with accepted formats and per-item/per-call limits. It is clearly separable from siblings like woodstitch_list_files, woodstitch_delete_file, and woodstitch_create_eval without opening any schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives an explicit downstream routing rule: 'Use an id anywhere woodstitch_create_eval takes an audio reference', which tells the agent exactly where the output flows next. It lacks an explicit when-not or a stated alternative (e.g., reusing an existing id from woodstitch_list_files instead of re-uploading), so it stops short of full 5-level guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 13 tool updatesv0.1.5
    • First observedwoodstitch_account
    • First observedwoodstitch_cancel_eval
    • First observedwoodstitch_create_eval
    • First observedwoodstitch_delete_file
    • First observedwoodstitch_export
    • First observedwoodstitch_get_eval
    • First observedwoodstitch_get_results
    • First observedwoodstitch_list_evals
    • First observedwoodstitch_list_files
    • First observedwoodstitch_pricing
    • First observedwoodstitch_start_eval
    • First observedwoodstitch_templates
    • First observedwoodstitch_upload_files

TDQS

A3.9/5.0

Scored across 13 tools

Disambiguation5/5

Each tool targets a distinct resource and action: informational lookups (templates, pricing), file management (upload/list/delete), and a clearly separated eval lifecycle (create/start/get/results/export/cancel). The potentially confusable pairs are explicitly differentiated — create_eval only prices while start_eval spends money, and get_eval returns live progress while get_results returns final metrics.

Naming Consistency4/5

All names share the woodstitch_ prefix and are snake_case, with verb_noun patterns for the operations (list_evals, upload_files, create_eval, cancel_eval). The three informational tools (templates, pricing, account) are bare nouns rather than verb_noun, which is a minor but sensible deviation since they take no action.

Tool Count5/5

Thirteen tools is well within the sweet spot and every tool maps to a real step in the workflow: learn the product, upload clips, price, start, monitor, collect results, export, cancel. Nothing looks redundant or padded.

Completeness4/5

The full lifecycle is covered — discovery, file CRUD, quoting, starting, polling, results, export, cancellation, and account balance — with no obvious dead ends. Minor gaps remain, such as no way to delete an eval record or re-inspect a stored eval's compiled design after creation (only re-pricing via dry_run).

Maintenance

ActivityMaintained
ResponsivenessNo recent community issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    A
    quality
    C
    maintenance
    Real human judgment as agent tools -- an AI agent can ask a question and get back a structured, schema-validated JSON answer from a real quality-scored human. 16 response types (yes/no, ratings, rankings, A/B tests, sentiment, image/video/audio review, voice/video/photo capture). Fully programmatic signup with a $5 free trial credit, no card required.
    7
    66 npm
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables agents to generate TTS audio, verify it with independent ASR, measure quality against gates, escalate retries, and receive structured QC reports to improve text.
    1
    MIT
  • A
    license
    Not graded
    quality
    C
    maintenance
    Enables AI assistants to obtain calibrated probabilities, option picks, scale scores, and multi-question judgments for batches of short text, returning numeric results for sorting, filtering, and counting.
    MIT
  • A
    license
    Not graded
    quality
    B
    maintenance
    Enables assistants to queue and monitor local language and vision model evaluations, compare results with confidence intervals, and retrieve routing recommendations for task categories.
    MIT