Skip to main content
Glama

checkd-mcp

An MCP server (stdio) that lets AI assistants use Check’d: read run results, AI reports and findings with their evidence, and, when you ask, start runs, re-run failed tests or scan a website.

Needs Node.js 20 or newer. Built on the official MCP TypeScript SDK and the checkd-cli core.

Credentials

  • CHECKED_API_KEY: a workspace API key (Settings, API keys in the app). Reaches that workspace only.

  • or CHECKED_TOKEN, or the sign-in saved by checked login on this machine.

  • CHECKED_API_URL: only if you use a server other than https://getchecked.twelveai.app.

Without a credential the server still starts; every tool answers with how to sign in.

Related MCP server: Autotest AI MCP Server

Add it

Claude Code:

claude mcp add checked --env CHECKED_API_KEY=tfy_your_key -- npx -y checkd-mcp

Cursor (.cursor/mcp.json, or ~/.cursor/mcp.json for every project):

{
  "mcpServers": {
    "checked": { "command": "npx", "args": ["-y", "checkd-mcp"], "env": { "CHECKED_API_KEY": "tfy_your_key" } }
  }
}

VS Code (.vscode/mcp.json; VS Code asks for the key once):

{
  "inputs": [{ "type": "promptString", "id": "checked-api-key", "description": "Check’d workspace API key", "password": true }],
  "servers": {
    "checked": { "type": "stdio", "command": "npx", "args": ["-y", "checkd-mcp"], "env": { "CHECKED_API_KEY": "${input:checked-api-key}" } }
  }
}

Tools

Read-only (free; readOnlyHint: true):

Tool

What it does

whoami

Which account or key, workspaces, plans

list_projects

Projects with their last run

list_environments

A project’s environments (secrets masked)

list_runs

Recent runs

get_run

Status, release gates, AI verdict, top problems with finding ids

wait_for_run

Waits up to 600 s for a run to finish

get_run_results

Individual API test results

get_run_report

The AI QA report

list_findings, get_finding

Failed tests and scan findings; full request, response, checks, stack trace, diagnosis and fix

list_issues, get_issue

The issue inbox: production errors and test findings, with the cached AI explanation (see the seam below)

get_usage

Session and month usage, and whether work is blocked

preview_import

What an API import would add; nothing saved

Start work or change things (readOnlyHint: false; descriptions say they use allowance):

Tool

What it does

start_run

Start a run; optional wait_seconds (max 600) returns the result

rerun_failed

Re-run only the failed tests of a finished API run

start_scan

Website scan (uses or creates the workspace’s project for that host)

import_spec

Save endpoints from an API document

cancel_run

Cancel a run

Each tool returns a readable text answer plus the JSON it was built from. Errors come back as isError results; a 402 says the allowance is used up, when it resets, and not to retry.

Resources: checked://runs/{run}/report, checked://runs/{run}/summary, checked://findings/{id}. Prompt: fix_failures (look at the latest run of a project and fix what it found).

The server sends instructions to the client: prefer read-only tools, start work only when asked, never loop on work-starting tools, and stop on a usage limit.

Issues seam

The issue tools use issuesProvider() from checkd-cli/core. When /cli/whoami reports capabilities.issuesInbox: true they read the issue inbox (get_issue includes the privacy-safe evidence, the fix pull request and the cached AI explanation; it never starts a new one); on older servers they read findings. list_findings and get_finding always read runs.

Development

npm install     # links ../cli; build it first (cd ../cli && npm run build)
npm test        # tool schemas and handlers against a mocked API, over an in-memory MCP transport
npm run build
CHECKED_API_KEY=tfy_... CHECKED_API_URL=http://localhost:4300 node dist/bin.js

Available Tools

19 tools
cancel_runCancel a runA
DestructiveIdempotent

Cancels a queued or running run. Stops further usage by that run.

ParametersJSON Schema
NameRequiredDescriptionDefault
runYesRun ref (e.g. payments-kqzxm) or numeric id

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare destructiveHint=true, readOnlyHint=false, and idempotentHint=true, so the safety profile is covered structurally. The description adds a genuine consequence beyond those flags—'Stops further usage by that run'—telling the agent what resource consumption ceases, which is the kind of 'what gets destroyed' context the annotations do not spell out. It still omits the outcome for a run that is already terminal.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences, zero filler, with the action and its scope front-loaded before the effect. Nothing could be trimmed without losing information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a single-parameter mutation with no output schema, the description plus annotations cover the essentials an agent needs to call it safely. The remaining gap is minor: it does not say what happens when the target run is already complete or unknown to the cancellation path.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%: the single 'run' parameter already documents that it accepts a run ref or numeric id. The description adds nothing about the parameter, so the baseline 3 applies—the schema does all the work.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource ('Cancels ... a run') and narrows the scope to 'queued or running' runs, which lets an agent tell it apart from start_run, rerun_failed, and wait_for_run by name alone. It stops short of explicitly contrasting itself with those siblings, so it is clear but not fully differentiated.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Applicability is implied by state: only a queued or running run can be cancelled. There is no explicit when-to-use guidance, no statement of what to do instead once a run has finished or failed, and no mention of the alternatives (rerun_failed, start_run) that an agent might confuse this with.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_findingGet findingA
Read-onlyIdempotent

Read-only. One finding with its full evidence: the exact request and response, failed checks, stack trace or console errors when captured, the diagnosis and how to fix it.

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesr123 or f456

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover safety (readOnly, idempotent, non-destructive, closed-world), so 'Read-only' is redundant, but the description adds real value by disclosing exactly what the response contains (raw request/response, failed checks, stack traces, console errors, diagnosis, fix). It omits behavior for a missing/invalid id.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences with the key classification ('Read-only') front-loaded and the content list kept dense and useful. Minor redundancy: 'Read-only' duplicates the readOnlyHint annotation.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the burden of describing return values and does so thoroughly, listing every evidence category an agent would receive. Only error/not-found behavior is left unexplained.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the single id parameter is documented in the schema ('r123 or f456'). The description adds no format or constraint detail beyond that, so baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('One finding with its full evidence'), and enumerates the payload contents, which clearly distinguishes it from the sibling list_findings. It does not name any sibling explicitly, so it stops short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The singular 'One finding' implies fetching details for a known id after a listing call, but there is no explicit when-to-use guidance or routing away from list_findings/get_issue. Usage is inferable but not stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_issueGet issueA
Read-onlyIdempotent

Read-only. One issue: occurrences, users, releases, the privacy-safe evidence (error, stack frames in the app’s code, breadcrumbs, request), its fix pull request, and the cached AI explanation (what is broken, likely cause, where to look, suggested fix). Never starts a new explanation. Also accepts finding ids (r123, f456).

ParametersJSON Schema
NameRequiredDescriptionDefault
idYesAn issue ref like kqzxmvbt, or r123 / f456

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent and non-destructive, so the bar is lower, and the description still adds real context: the cached-vs-generated distinction ('Never starts a new explanation') is a genuine side-effect boundary, and 'privacy-safe evidence' tells the agent what class of data comes back. It stops short of describing pagination or size limits.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Front-loaded with the mutation-safety fact and the single-resource scope, then the payload list. It is one dense paragraph with no filler sentences, though the mid-sentence enumeration of evidence types is long enough to slightly dilute the key routing information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the return-value burden and does so by listing the returned fields, including the AI explanation's sub-parts. Annotations cover the safety profile. It is nearly complete, missing only volume/shape caveats (pagination, evidence truncation).

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the single 'id' parameter already documents the ref format including r123/f456. The description's 'Also accepts finding ids' restates that schema detail rather than adding new syntax or constraints, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The first clause names a specific verb and resource ('Read-only. One issue') and then enumerates the payload: occurrences, users, releases, evidence, fix PR, cached AI explanation. This clearly separates it from list_issues and get_finding, and the 'Also accepts finding ids (r123, f456)' note pins down its scope relative to the finding siblings.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It gives an important when-not ('Never starts a new explanation'), which prevents an agent from expecting generation on this call. However it never explicitly routes the agent to alternatives (e.g. get_finding for a finding-centric read, or why to pick this over that) despite overlapping id acceptance with get_finding, leaving the sibling choice to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_runGet run resultA
Read-onlyIdempotent

Read-only. A compact result for one run: status and counts, release gates, the AI verdict and the top problems with ids for get_finding.

ParametersJSON Schema
NameRequiredDescriptionDefault
runYesRun ref (e.g. payments-kqzxm) or numeric id

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, and destructiveHint/openWorldHint false, so 'Read-only' is partly redundant. The value-add is the return-shape disclosure and the word 'compact', which signals a summarized rather than exhaustive payload — meaningful given there is no output schema.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence plus a safety lead-in, front-loaded and with zero filler. Every clause either qualifies safety or names returned content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With one fully documented parameter and no output schema, the description does the necessary work of sketching the return payload. It is slightly short on distinguishing this compact view from get_run_results/get_run_report, but nothing required to invoke the call correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the single 'run' parameter is documented in the schema with accepted forms (slug or numeric id). The description adds nothing beyond that, so the baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Get run', 'a compact result for one run') and enumerates the payload (status, counts, release gates, AI verdict, top problems). It does not, however, differentiate itself from the close siblings get_run_results and get_run_report, leaving the agent to guess which run-scoped reader to pick.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The phrase 'ids for get_finding' is an explicit hand-off to a sibling, which is genuine routing guidance. But there is no statement of when to use this over list_runs, get_run_results, get_run_report or wait_for_run, so the when-to-use decision is only implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_run_reportGet AI reportA
Read-onlyIdempotent

Read-only. The AI QA report for a run: verdict (ship, ship with care, hold), findings with impact and fixes, next steps, what was not tested.

ParametersJSON Schema
NameRequiredDescriptionDefault
runYesRun ref (e.g. payments-kqzxm) or numeric id

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover readOnly/idempotent/non-destructive, so 'Read-only' is partly redundant, but the description earns credit by disclosing the return shape that no output schema provides: the verdict vocabulary (ship, ship with care, hold), findings carrying impact and fixes, and explicit 'what was not tested' coverage. It omits whether the report may be absent or still generating, which is a real behavioral gap given wait_for_run exists as a sibling.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single compact sentence set, front-loaded with the safety note and then the payload contents; every clause carries information. The leading 'Read-only.' duplicates the annotation and is the only real waste.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the return-value burden and does so well by naming the verdict enum and finding structure. For a one-parameter read tool with full annotations this is nearly complete; the missing piece is availability semantics — whether a report always exists or may need wait_for_run first.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The single 'run' parameter is fully documented in the schema (coverage 100%), including the ref-or-numeric-id format and examples. The description adds nothing about parameter semantics, so the baseline 3 for high schema coverage applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific resource ('the AI QA report for a run') and enumerates its contents — verdict, findings with impact and fixes, next steps, untested areas — which cleanly separates it from raw-data siblings like get_run_results and list_findings. It stops short of explicitly naming or contrasting those siblings, so an agent must infer the boundary itself.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the resource and the enumerated contents: fetch this when you want the synthesized verdict rather than raw results. However, the description never states when to prefer this over get_run_results, list_findings, or get_run, nor whether a report is guaranteed to exist for a run — the reader has to infer all of that.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_run_resultsGet test resultsB
Read-onlyIdempotent

Read-only. Individual API test results of a run; filter to failures with status "failed,errored".

ParametersJSON Schema
NameRequiredDescriptionDefault
runYesRun ref (e.g. payments-kqzxm) or numeric id
limitNo
statusNo

TDQS

B3.4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint and destructiveHint=false, so the leading 'Read-only' partly repeats structured data. It does add one genuinely useful behavioral detail the annotations lack — the comma-separated multi-value syntax for the status filter — but says nothing about pagination despite a limit parameter, or about result ordering/volume.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short, front-loaded fragments with no filler; the resource and the filtering trick are both immediate. The leading 'Read-only' is slightly wasteful given annotations, but nothing else is padded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, so the description should carry return-value expectations; 'individual API test results' is a thin sketch of the payload. With an undocumented limit parameter and no pagination or shape guidance, it is adequate but leaves real gaps for a results-listing tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 33%; 'run' is documented in the schema while 'limit' and 'status' are not. The description partially compensates by showing the status value syntax ('failed,errored'), but limit remains undocumented in both places, so it neither fully compensates nor is redundant.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: reads individual API test results belonging to a named run. This is distinct from get_run_report (aggregate) and list_findings (issue-level), though the description never names those siblings to make the distinction explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The filter hint ('filter to failures with status "failed,errored"') implies the main use case of narrowing to failures, but it never states when to prefer this tool over get_run_report, get_run, or list_findings, nor any exclusions.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

get_usageGet usageB
Read-onlyIdempotent

Read-only. How much of this session’s and this month’s allowance the workspace has used, and whether new work is blocked.

ParametersJSON Schema
NameRequiredDescriptionDefault
workspaceNo

TDQS

B3.3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, and destructiveHint=false, so 'Read-only' merely restates structured data. The description does add real behavioral content by naming what is reported (session vs. month consumption and blocked status), but omits any note on scoping, caching, or refresh behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two compact sentences with the payload information front-loaded. The leading 'Read-only.' fragment is redundant against the annotations, a small amount of waste in an otherwise efficient description.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description carries the burden of describing returns and does so adequately — session and monthly allowance consumption plus a blocked flag. Units, exact fields, and reset semantics are left unspecified, but the shape of the answer is conveyed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% and the sole 'workspace' parameter is never mentioned in the description. The phrase 'the workspace has used' weakly implies the default workspace, but it provides no guidance on what value to pass or what happens when omitted.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific informational resource (allowance/usage) and its scope (this session and this month, plus whether new work is blocked). It's clear what the tool returns, though it makes no attempt to differentiate from siblings — reasonable here since no sibling is a usage/quota tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit when-to-use or when-not-to-use guidance, but 'whether new work is blocked' implies the natural context of checking quota before starting a run. Usage is inferable rather than stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

import_specImport an API specA
Destructive

Saves endpoints into an API project from an OpenAPI, Postman or other document (merged by method and path; replace swaps the whole list). Call preview_import first and confirm with the user. May start an AI reading of the API, which uses allowance.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlNo
contentNo
projectYesProject ref (e.g. payments-kqzxm) or numeric id
replaceNo

TDQS

A4.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already flag destructiveHint=true, openWorldHint=true and idempotentHint=false. The description adds real value beyond them by explaining the mutation model ("merged by method and path; replace swaps the whole list") and a cost side-effect ("May start an AI reading of the API, which uses allowance"). It stops short of stating permissions, reversibility or rate limits, so it is strong but not exhaustive.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight clauses, all front-loaded: what it does, how merging works, then the required prerequisite. No filler or restated name/title.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a destructive import with no output schema and annotated write behavior, the description covers the merge model, the safety workflow and the cost side-effect. Only the url/content parameter distinction and any failure/partial-import behavior are left unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 25% — just `project` carries a description. The description clarifies the semantic effect of `replace` ("swaps the whole list") and implies url/content by naming source document types, but does not disambiguate `url` vs `content` or address the 8MB content cap and merge-vs-replace interplay with the other params. Baseline 3 given partial schema help.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ("Saves endpoints into an API project") plus the accepted source formats (OpenAPI, Postman, other documents). The merge-vs-replace clause and the preview_import prerequisite make it clearly distinguishable from its sibling preview_import.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines5/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Explicitly prescribes the workflow: "Call preview_import first and confirm with the user." It names the alternative tool and the gating condition for proceeding, leaving no inference required.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_environmentsList environmentsA
Read-onlyIdempotent

Read-only. Lists a project’s environments (base URL, auth type). Secrets are masked.

ParametersJSON Schema
NameRequiredDescriptionDefault
projectYesProject ref (e.g. payments-kqzxm) or numeric id

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, and destructiveHint=false, so the leading "Read-only" adds nothing new. However, "Secrets are masked" is genuine behavioral context about the response that annotations do not convey, warning the agent that credential values will not be exposed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three compact clauses with no filler, and the safety/scope constraint is front-loaded. Every clause contributes distinct information: read-only, the resource and its returned fields, and the masking behavior.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description usefully names the returned fields (base URL, auth type) and the masking behavior. It does not mention ordering, pagination, or error behavior when the project ref is invalid, leaving a small gap for a listing tool.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% and the single parameter is fully documented as a project ref or numeric id, so the schema carries the weight. The description only implies the parameter's role ("a project's environments") without adding format or constraint detail, matching the baseline for fully-covered schemas.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Lists) and resource (a project's environments), and even names the fields returned (base URL, auth type). The resource is naturally distinct from siblings like list_projects or list_runs, but the description never explicitly contrasts with an alternative tool, which keeps it short of a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied: the tool scopes to "a project's environments" and requires a project ref, so an agent can infer it is the lookup for environments of a known project. There is no explicit when-to-use, when-not-to-use, or named alternative, so this is minimum-viable rather than strong guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_findingsList findingsA
Read-onlyIdempotent

Read-only. Problems from a run (default the latest finished one): failed API tests and website scan findings, worst first.

ParametersJSON Schema
NameRequiredDescriptionDefault
runNoDefault: the latest finished run
limitNo
projectYesProject ref (e.g. payments-kqzxm) or numeric id
severityNoThis severity and worse

TDQS

A3.8/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already cover the safety profile (readOnly, idempotent, non-destructive), and 'Read-only' merely repeats that. The description does add genuinely new behavioral context: the default run selection, the result ordering ('worst first'), and the kinds of records returned.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences, front-loaded with the read-only constraint, then scop; no filler and every clause carries information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 4-parameter, read-only query tool with no output schema, the description covers what is returned and in what order. It omits limit/pagination behavior and filtering semantics, which an agent would still need to infer from the schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 75%, so most parameters are already documented, including the run default which the description merely restates. The description adds no format or syntax detail for `severity`, `limit`, or how `project` refs/hashes are accepted.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a clear verb+resource ('list findings') and defines the content precisely: failed API tests and website scan findings from a run. It implicitly distinguishes findings (problems within a run) from the sibling list_issues/get_issue, though it never names those alternatives explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage context is implied by 'default the latest finished one' and the run-scoped framing, but there is no explicit when-to-use guidance, no exclusion of list_issues, and no pointer to get_finding for single-item retrieval.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_issuesList issuesA
Read-onlyIdempotent

Read-only. Open issues for a project from the issue inbox: errors real users hit in production and findings from tests and scans, highest impact first, with the one-line AI summary when one exists. On servers without the inbox, the latest run’s findings.

ParametersJSON Schema
NameRequiredDescriptionDefault
runNoDefault: the latest finished run
limitNo
sourceNo
statusNoDefault open (new, regressed, investigating)
projectYesProject ref (e.g. payments-kqzxm) or numeric id
severityNoThis severity and worse

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent and non-destructive, so the safety profile is covered. The description adds genuinely new behavioral context beyond that: the sort order (highest impact first), the inclusion of a one-line AI summary when present, and a fallback data source when the inbox is unavailable. It stops short of documenting pagination or the limit parameter's effect.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences with the 'Read-only' qualifier front-loaded and the core purpose immediately following. The fallback clause is placed last, which is correct emphasis. Nothing is redundant, though the embedded clause listing sources makes the first sentence slightly dense.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, the description does the right work of summarizing returned content (issues with AI summaries, impact-ordered) and noting the fallback source. Given six parameters with one required, an agent has enough to call this correctly, though return cardinality and pagination are unaddressed.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 67%, so several parameters (limit, source, severity) are already documented structurally. The description reinforces the default open-status filter implicitly ('open issues') but adds no syntax or format detail beyond the schema. This matches the baseline 3 when the schema does most of the work.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific verb and resource ('Open issues for a project from the issue inbox') and qualifies the scope ('errors real users hit in production and findings from tests and scans'). It implicitly differentiates from the raw-findings sibling by describing an aggregated, prioritized inbox view, but never names list_findings explicitly, so sibling routing is left partly to inference.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the scope ('open issues... from the issue inbox') but there is no explicit when-to-use/when-not guidance and no named alternative among siblings like list_findings or get_issue. The one routing hint is the fallback behavior on inbox-less servers, which addresses environment rather than task selection.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_projectsList projectsB
Read-onlyIdempotent

Read-only. Lists Check’d projects (API projects test endpoints; website projects are scanned in a browser) with their last run.

ParametersJSON Schema
NameRequiredDescriptionDefault
workspaceNoWorkspace slug or id; default all the credential can see

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so 'Read-only.' is largely redundant. The description does add genuine behavioral context by disclosing that results include each project's last run and by explaining the API-vs-website project distinction, but it says nothing about pagination, ordering, or the default 'all workspaces' scope behavior.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight clauses with no wasted words and the resource up front. The leading 'Read-only.' duplicates the annotations rather than earning its place, a minor inefficiency.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a simple read-only list tool with a fully documented single optional parameter and no output schema, the description covers the key question of what comes back (projects plus last run). Additional detail on result shape or volume is not strictly required here.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100% for the single workspace parameter, so the schema already documents it fully (slug or id, defaults to all visible). The description adds nothing about the parameter, which is the expected baseline when the schema does the work.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (Lists) and resource (Check'd projects) and even distinguishes two project subtypes (API vs website) with an important semantic difference in how they are exercised. It does not compare itself to sibling list tools like list_runs or list_environments, so it falls just short of 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The description says what the tool returns but gives no when-to-use guidance, no condition for choosing it over siblings such as list_runs or whoami, and no exclusions. Usage must be inferred from the name alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

list_runsList runsC
Read-onlyIdempotent

Read-only. Recent runs of a project, newest first.

ParametersJSON Schema
NameRequiredDescriptionDefault
limitNo
statusNoComma separated: queued, running, passed, failed, errored, cancelled
projectYesProject ref (e.g. payments-kqzxm) or numeric id

TDQS

C2.7/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, non-destructive, and closed-world, so the 'Read-only' prefix is largely redundant. The description's only added behavioral fact is that results are ordered newest first, which is genuinely useful context not present in the annotations or schema, but defaults for limit and any pagination behavior are undisclosed.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short fragments, front-loaded with the safety profile followed by scope and ordering, with zero filler. It is efficient, though the terseness borders on under-specification rather than optimal conciseness.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a three-parameter list tool with no output schema, an agent still lacks the default and maximum for limit, the accepted status values (only in the schema property, not the description), and whether results are paginated or truncated. Annotations cover the safety profile, but the operational details an agent needs to call this correctly are missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is only 67%: project and status carry descriptions, but limit has none in either schema or description, and its default value is never stated. The description adds nothing about which parameters exist or how to combine them (e.g. that status accepts a comma-separated list and limit caps at 50), so it does not compensate for the coverage gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose3/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names the resource (runs) and scopes it to a single project, with ordering ('newest first'), so the agent can tell it is a listing tool rather than a single-record fetcher like get_run. However, it never states the verb explicitly and gives no contrast against siblings such as get_run_results or get_run_report, leaving differentiation to inference.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use guidance, no stated prerequisites, and no named alternative. The phrase 'recent runs' hints at a browsing use case, but nothing tells the agent when to prefer this over list_findings, get_run, or wait_for_run. No explicit exclusion of cases where filtering by status or a single run is more appropriate.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

preview_importPreview an API importA
Read-onlyIdempotent

Read-only (nothing is saved). Parses an OpenAPI, Swagger, Postman, Insomnia, HAR, GraphQL or cURL document, given as text or a URL, and says what an import into the project would add.

ParametersJSON Schema
NameRequiredDescriptionDefault
urlNoOr a URL to fetch it from
contentNoThe document text
projectYesProject ref (e.g. payments-kqzxm) or numeric id

TDQS

A4/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, openWorldHint and destructiveHint=false, so the safety profile is covered. The description's "nothing is saved" usefully reinforces that no state is mutated and implies network fetching for the URL form, but adds nothing about failure modes, size limits, or partial-parse behavior beyond what structured fields give.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two tight sentences with zero waste; the most decision-relevant fact (read-only, nothing saved) is front-loaded before the format list and outcome.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a read-only, fully-annotated tool with complete parameter documentation and no output schema, the description covers what it does and what it returns at a summary level. It could be stronger by contrasting with import_spec or noting the response shape, but nothing essential is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so the schema documents url, content and project itself. The description's "given as text or a URL" only restates the two input modes already encoded in the schema, adding no new syntax, exclusivity rule, or format guidance.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (preview/parse) and resource (OpenAPI/Swagger/Postman/Insomnia/HAR/GraphQL/cURL document) and immediately scopes it as non-persisting ("Read-only (nothing is saved)"). The phrase "says what an import into the project would add" cleanly separates it from the sibling import_spec without needing the schema.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The "nothing is saved" framing and "what an import would add" make the use case clear: dry-run before importing. However, it never names the alternative (import_spec) or states an explicit when-not-to-use condition, so the routing to the write-side sibling is left to inference.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

rerun_failedRe-run failed testsA

Re-runs only the failed and errored tests of a finished API run, in place, to confirm a fix. Uses the workspace’s usage allowance (session and month).

ParametersJSON Schema
NameRequiredDescriptionDefault
runYesRun ref (e.g. payments-kqzxm) or numeric id

TDQS

A4.2/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations declare the mutation profile (readOnlyHint=false, idempotentHint=false, destructiveHint=false), and the description adds two things they don't carry: the operation is 'in place' (existing run is modified) and it consumes the workspace's session and monthly usage allowance. It does not say what happens to prior results or whether the run becomes reusable afterward.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single sentence that leads with the action and scope, then adds the quota caveat. No filler or restated title content.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a one-parameter mutation tool with no output schema, the description covers scope, intent, in-place semantics, and billing impact. Minor gaps remain around return value and effect on the original run's results, but nothing an agent needs to invoke it correctly is absent.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Only one parameter ('run') and schema description coverage is 100%, so the schema already explains the run-ref-or-numeric-id format. The description adds no parameter-level detail, which is the expected baseline when the schema does the work.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb ('Re-runs') and a precisely scoped resource ('only the failed and errored tests of a finished API run'). The scope words 'only', 'failed and errored', and 'finished' implicitly separate it from start_run, which begins a fresh run.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives a clear triggering context, 'to confirm a fix', so an agent knows when this is the right tool rather than re-starting the run. It does not name start_run as the alternative or state when not to use it, so it stops short of full routing guidance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

start_runStart a test runA

Starts a run on a project: API tests for API projects, a website scan (or AI exploration) for website projects. Uses the workspace’s usage allowance (session and month). Optionally waits for the result (at most 600 s); otherwise use wait_for_run or get_run.

ParametersJSON Schema
NameRequiredDescriptionDefault
kindNosuite = API tests (default for API projects); frontend_health = website scan (default for website projects)
depthNoAPI runs: critical is fastest and cheapest; deep (default) runs every test
projectYesProject ref (e.g. payments-kqzxm) or numeric id
max_pagesNo
start_urlNoWebsite runs: the page to start from
suite_idsNo
environmentNoEnvironment id or name; default is the project’s default
wait_secondsNoWait for the result up to this long (default 0: return at once)

TDQS

A3.9/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

With annotations already declaring non-read-only, open-world, non-idempotent mutation, the description adds genuinely new behavioral context: the run consumes the workspace's usage allowance (session and month) and the optional wait is capped at 600 s. It stops short of stating what a run costs, whether concurrent runs are allowed, or that a run id is produced.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences with the core action front-loaded, then consumption cost, then the wait/hand-off pattern. No filler, though the parenthetical '(or AI exploration)' is the only hint that frontend_explore/frontend_flow kinds exist.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness4/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a mutation tool with no output schema and openWorld=true, the description covers what triggers the run, its side effect on quota, and how to retrieve the result via siblings. It leaves the return shape (run id) and permission requirements implicit, but the routing to wait_for_run/get_run covers the main gap.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 75%, so the schema largely carries the parameter burden. The description maps project type to the suite/frontend_health kinds and bounds wait_seconds, but says nothing about depth, max_pages, start_url, suite_ids, or environment beyond what the schema already documents. Baseline 3 is appropriate.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource ('Starts a run on a project') and explains what a run means per project type — API tests, website scan, or AI exploration. It does not, however, distinguish itself from the sibling start_scan, which an agent could easily confuse with the website-scan branch of this tool.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines4/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Gives explicit routing for result retrieval: 'Optionally waits for the result (at most 600 s); otherwise use wait_for_run or get_run.' The kind/scope defaults per project type are also implied. No exclusion guidance against the sibling start_scan, and no prerequisites (auth/permissions) are stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

start_scanStart a website scanA

Scans a website for broken links, console and network errors, accessibility and speed. Uses the workspace’s website project for that host, creating one if needed. Uses the workspace’s usage allowance (session and month).

ParametersJSON Schema
NameRequiredDescriptionDefault
urlYes
max_pagesNo
workspaceNo
wait_secondsNo

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare non-read-only, non-idempotent, open-world behavior. The description adds meaningful context beyond that: it consumes the workspace's usage allowance (session and month) and auto-creates a project if needed. That is real behavioral disclosure – cost and side effects – though it omits whether the call blocks, returns a run id, or how long it takes.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three tight sentences, front-loaded with what is scanned. Every sentence carries information; nothing is padded.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

A non-idempotent, quota-consuming scan tool with no output schema and zero parameter documentation. The description covers side effects and quota, which is valuable, but leaves an agent unable to predict return shape (run handle vs inline results) or the role of wait_seconds/max_pages.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0%, so the description must compensate for all 4 undocumented parameters (url, max_pages, workspace, wait_seconds). It mentions 'that host' and 'workspace' implicitly but explains none of them: no crawl limit semantics for max_pages, no meaning for wait_seconds, no format guidance for url. This is a major gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (scans) and resource (website), and enumerates exactly what it checks: broken links, console and network errors, accessibility, and speed. Clearly distinct from sibling run tools like start_run or rerun_failed, which operate on test runs rather than website scans.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies usage by describing what a scan does and mentioning workspace project resolution, but never says when to prefer this over start_run or how it relates to wait_for_run/get_run_results. No explicit exclusions or alternative routing.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

wait_for_runWait for a runA
Read-onlyIdempotent

Read-only. Waits (at most 600 s) for a queued or running run to finish, then returns its result. Does not start anything.

ParametersJSON Schema
NameRequiredDescriptionDefault
runYesRun ref (e.g. payments-kqzxm) or numeric id
timeout_secondsNoDefault 300

TDQS

A3.7/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare read-only and idempotent, but they say nothing about blocking. The description adds the critical behavioral trait that this call blocks up to 600 s and never triggers work, which an agent cannot infer from the annotations. It omits what happens if the wait expires (error vs. partial result), keeping it short of a 5.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Three short, front-loaded sentences with zero filler; the blocking constraint and the non-starting guarantee are both stated up front. 'Read-only' slightly duplicates the annotation but costs almost nothing.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

No output schema exists, so the description is the only place return behavior could be explained, and 'returns its result' is vague about what that result contains. Similarly, timeout-expiry behavior is unstated. It is adequate for calling the tool but incomplete for interpreting the outcome.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 100%, so both parameters (run ref/numeric id, timeout_seconds with default 300 and max 600) are fully documented in the schema. The description restates the 600 s cap but adds no new semantics for the run reference. Baseline 3 applies.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb (waits) and resource (run), plus an explicit negative boundary ('Does not start anything') that separates it from start_run/rerun_failed. It does not explicitly name get_run, the obvious non-blocking sibling, but the blocking semantics make the distinction inferable.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Implies the use case (you want to block on a queued or running run rather than poll) but never states when to prefer this over get_run or list_runs polling. No exclusions or alternatives are given, so usage is only implied.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

whoamiWho am IA
Read-onlyIdempotent

Read-only. Shows which Check’d account or workspace API key this server uses, its workspaces and plans.

ParametersJSON Schema
NameRequiredDescriptionDefault

No parameters

TDQS

A4.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false, and openWorldHint=false, so 'Read-only' in the description is a restatement rather than added value. It does add the scope of the returned identity (account/workspace API key, workspaces, plans), which is useful context, but it omits any note on auth requirements or throttling. With rich annotations covering the safety profile, this is adequate but thin.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences, safety attribute front-loaded, then the substance of what is returned. No filler, nothing an agent has to read twice.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness5/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a zero-parameter diagnostic with no output schema, the description is the only place to learn what comes back, and it names it precisely: the account or workspace API key in use, plus its workspaces and plans. Nothing needed to call it correctly is missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters4/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

The tool takes zero parameters, so the baseline is 4. The description correctly adds nothing about argument syntax because there is none, and it does not mislead about inputs.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose5/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description gives a specific verb and resource: it 'shows which Check'd account or workspace API key this server uses, its workspaces and plans.' That is well beyond restating the name 'whoami' and it clearly distinguishes this tool from the list_*/get_* siblings, which enumerate issues, runs, and findings rather than identity/credential scope.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no explicit when-to-use statement, no prerequisites, and no named alternatives. The 'whoami' convention makes the purpose inferable and there is no competing tool that overlaps, but the description leaves the agent to infer that this is the right call for verifying which key/account is active. Usage is implied, not stated.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 19 tool updatesv0.1.0
    • First observedcancel_run
    • First observedget_finding
    • First observedget_issue
    • First observedget_run
    • First observedget_run_report
    • First observedget_run_results
    • First observedget_usage
    • First observedimport_spec
    • First observedlist_environments
    • First observedlist_findings
    • First observedlist_issues
    • First observedlist_projects
    • First observedlist_runs
    • First observedpreview_import
    • First observedrerun_failed
    • First observedstart_run
    • First observedstart_scan
    • First observedwait_for_run
    • First observedwhoami

TDQS

B3.4/5.0

Scored across 19 tools

Disambiguation4/5

Most tools have clearly distinct resources/actions (projects, environments, runs, findings, issues, imports, scans, usage). A few could be confused—start_run vs start_scan on website projects, and list_issues/list_findings/get_issue/get_finding around problem retrieval—but the detailed descriptions explain the differences.

Naming Consistency4/5

All tool names are snake_case and mostly follow a verb_noun pattern (list_projects, get_run, start_run, cancel_run). Minor deviations such as whoami, wait_for_run, and rerun_failed keep it from perfect consistency.

Tool Count3/5

19 tools is in the heavy 16-25 range for a single integration. While most map to distinct entities, several granular run/finding retrieval tools (get_run_results, get_run_report, list_findings, get_finding) could be consolidated.

Completeness3/5

Core operational workflows are covered: inspect projects/environments/runs/findings/issues, import specs, start/rerun/cancel runs, scan websites, and check usage. However there are notable gaps in setup/admin—no create/update/delete for projects or environments, and issue/finding tools are read-only—so onboarding a new API project or editing configuration is a dead end.

Maintenance

ActivityMaintained
ResponsivenessNo issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    B
    quality
    A
    maintenance
    Enables AI assistants to interact with SmartBear's suite of testing and monitoring tools including BugSnag, Swagger, Reflect, and more through natural language, allowing querying of testing data, analysis of performance metrics, and management of test automation.
    309
    12,469 npm
    44
    MIT
  • A
    license
    Not graded
    quality
    D
    maintenance
    Provides AI assistants with seamless access to SmartBear's suite of testing and monitoring tools, enabling querying testing data, analyzing performance metrics, and managing test automation through natural language.
    12,469 npm
    MIT
  • A
    license
    A
    quality
    B
    maintenance
    Lets an assistant read recorded production AI agent runs — steps, model and tool calls with arguments, and outcomes — along with the findings of comparison runs that checked a prompt or model change against those recordings, and verify a signed evidence bundle offline with no account or network. It is strictly read-only: no tool starts a replay or comparison, so nothing can spend money or act without a person deciding.
    5
    36 PyPI
    Apache 2.0