Skip to main content
Glama
mcp-protocol-conformance

mcp-protocol-conformance

Part of the StudioMeyer MCP Stack โ€” Built in Mallorca ๐ŸŒด ยท โญ if you use it

mcp-protocol-conformance

npm version npm downloads License Last commit GitHub stars

This is a Foundation build of the StudioMeyer MCP Factory: every other Factory build runs through this harness before promotion (npm publish, marketplace submit, upstream PR).

A note from us

We have been building tools and systems for ourselves for the past two years. The fact that this repo is small and has few stars is not because it is new. It is because we only just decided to share what we have built. It is not a fresh experiment, it is a long story with a recent commit.

We love building things and sharing them. We do not love social media tactics, growth hacks, or chasing stars and followers. So this repo is small. The code is real, it gets used, issues get answered. Judge for yourself.

If it helps you, sharing, testing, and feedback help us. If it could be better, an issue is more useful. If you build something with it, tell us at hello@studiomeyer.io. That genuinely makes our day.

From a small studio in Palma de Mallorca.

Related MCP server: MCP Test Server

Install

npm install --save-dev mcp-protocol-conformance

The package ships both a CLI (mcp-conformance) and a TypeScript library entry (import ... from "mcp-protocol-conformance").

CLI usage

Run against a stdio server

mcp-conformance run \
  --target stdio \
  --cmd node \
  --cmd-arg dist/server.js \
  --spec 2025-06-18 \
  --suite all \
  --format terminal

Run against an HTTP server (Streamable HTTP, 2025-03-26+)

mcp-conformance run \
  --target http \
  --url https://memory.studiomeyer.io/mcp \
  --header "Authorization:Bearer ${TOKEN}" \
  --spec 2025-06-18 \
  --suite full \
  --format json --out report.json

Run only a subset of suites

mcp-conformance run --target stdio --cmd ./server --spec 2025-06-18 \
  --suite jsonrpc,version,schema

Compare two manifests

mcp-conformance compare \
  --expected manifests/v1.json \
  --actual   manifests/v2.json

Assert no breaking changes between two reports

mcp-conformance assert-no-breaking \
  --baseline reports/main.json \
  --current  reports/pr-42.json

Exit codes: 0 clean, 1 failures, 2 invocation error.

Library usage

import {
  runFullSuite,
  generateReport,
} from "mcp-protocol-conformance";

const report = await runFullSuite(
  { kind: "stdio", cmd: "node", args: ["dist/server.js"] },
  "2025-06-18",
  { suite: "all" },
);

console.log(generateReport(report, "terminal"));
if (report.status === "fail") process.exit(1);

MCP server usage

The harness is itself an MCP server. Start it over stdio and any MCP client can call its 12 tools:

node dist/server.js

Tools (all read-only, all destructiveHint: false):

#

Tool

Purpose

1

runJsonRpcCompliance

JSON-RPC 2.0 error-code matrix + response-envelope (result/error mutual exclusivity)

2

runSpecVersionAssertion

Verify advertised protocolVersion

3

runTransportSuite

Transport-layer ping + session-id

4

runOauthPkceFlow

OAuth 2.1 PKCE S256 (mock-AS or real-tenant)

5

runToolSchemaValidation

inputSchema (+ 2025-11-25 outputSchema / title) is valid JSON-Schema

6

runCapabilityIntrospection

initialize.capabilities matches behaviour (+ 2025-11-25 tasks)

7

runRoundtripSmoke

One tools/call per advertised tool

8

runAnnotationsAudit

readOnlyHint / destructiveHint hygiene

9

runFullSuite

All suites + summary

10

generateReport

Render JUnit / JSON / terminal

11

compareManifests

Diff two tool manifests

12

assertNoBreakingChanges

Diff two FullReports

Compatibility matrix

2024-11-05

2025-03-26

2025-06-18

2025-11-25

jsonrpc

yes

yes

yes

yes

version (handshake)

yes

yes

yes

yes

transport (stdio)

yes

yes

yes

yes

transport (http)

n/a

yes

yes

yes

oauth (mock)

n/a

yes

yes

yes

oauth (real)

n/a

yes

yes

yes

schema

yes

yes

yes

yes

capability

yes

yes

yes

yes

smoke

yes

yes

yes

yes

annotations

warn-only

warn-only

yes

yes

tasks (capability)

n/a

n/a

n/a

yes

tool title/output

n/a

n/a

n/a

warn-only

yes = suite runs and produces actionable results. n/a = capability not in spec; suite skips automatically. warn-only = suite runs but the spec does not formally require the feature.

The 2026-07-28 RC (stateless core, SEP-2575/2567) is intentionally not here โ€” it ships final on 2026-07-28; conformance for it lands in a later release against the final SDK.

Integration in Factory builds

In every Factory build's package.json:

{
  "scripts": {
    "factory:conformance": "mcp-conformance run --target stdio --cmd 'node dist/server.js' --spec 2025-06-18 --suite full"
  }
}

In CI:

- run: npm run factory:conformance

Spec references

About StudioMeyer

StudioMeyer is an AI and design studio based in Palma de Mallorca, working with clients worldwide. We build custom websites and AI infrastructure for small and medium businesses. Production stack on Claude Agent SDK, MCP and n8n, with Sentry, Langfuse and LangGraph for observability and an in-house guard layer.

License

MIT โ€” Copyright (c) 2026 Matthias Meyer (StudioMeyer)

Available Tools

12 tools
assertNoBreakingChangesB
Read-onlyIdempotent

Compare a baseline FullReport against a current FullReport. Returns a list of breaking changes and warnings.

NameRequiredDescriptionDefault
currentNo
baselineNo

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false and a closed world, so safety is covered. The description adds that the result is a list of breaking changes plus warnings, which is genuinely useful given there is no output schema, but it leaves a key ambiguity: despite the name 'assert', it is unclear whether breaking changes cause a failure/throw or are only reported.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences with the action and the return value front-loaded and no filler. Brevity is fine; the gaps are omissions of content rather than wasted wording.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness3/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a two-input comparison tool with no output schema and no annotations gap, the description covers the essentials but omits how to obtain FullReports, the result's structure beyond 'list', and the failure semantics implied by 'assert'. These are the details an agent would actually need to chain this tool correctly.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters3/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema coverage is 0% and the properties are untyped empty objects, so the description carries the burden of explaining the parameters. It does establish that both 'baseline' and 'current' are FullReport objects and what roles they play, but it says nothing about ordering, shape, or why required is empty when both inputs are obviously needed.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource: compare a baseline FullReport against a current FullReport and return breaking changes and warnings. That is clear enough to act on, but it never distinguishes itself from the sibling compareManifests (or generateReport, which presumably produces the FullReports), so an agent cannot tell from the description alone which comparison tool to pick.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use guidance, no exclusion of alternatives, and no mention of where the two FullReport inputs come from. An agent must guess whether this belongs in a release gate, a pre-merge check, or after generateReport.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

compareManifestsB
Read-onlyIdempotent

Diff two tool manifests โ€” added/removed/changed tools.

NameRequiredDescriptionDefault
actualYes
expectedYes

TDQS

B3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, non-destructive and closed-world, so the safety profile is covered structurally. The description adds the useful fact that results are bucketed into added/removed/changed, but says nothing about comparison depth (which fields are compared โ€” title, description, inputSchema, _meta?), whether tool order matters, or how nested schema differences are reported.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single compact sentence that front-loads the operation and the resource, with the output categories appended. Nothing redundant.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, nested object parameters, and zero schema description coverage, the description is too thin for the tool's complexity: return shape (counts? per-tool detail? paths to changed fields?) and diff direction are both left to inference. The core promise is clear but the operational details an agent needs to use the result are missing.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the two required parameters are deep nested manifest objects, so the description carries the burden โ€” and it doesn't. Critically, it never establishes the direction of the diff: whether "added" means present in actual but absent from expected. An agent can guess that expected is the baseline, but the description does not say so.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ("Diff") on a specific resource ("two tool manifests") and enumerates the result categories (added/removed/changed), so an agent can tell this is a comparison utility rather than a test runner like the sibling run* tools. It stops short of naming a sibling to distinguish itself from, notably assertNoBreakingChanges, which sounds closely related.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no statement of when to reach for this versus alternatives, despite a sibling (assertNoBreakingChanges) that plausibly performs an overlapping manifest comparison. No prerequisites, no note about whether the tool fails on differences or merely reports them.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

generateReportC
Read-onlyIdempotent

Render a FullReport as JUnit XML, JSON, or human-readable terminal output.

NameRequiredDescriptionDefault
formatYes
reportNo

TDQS

C2.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false and closed-world, so the safety profile is fully covered. The description reinforces that this is a pure transformation into one of three presentation formats, which is mild added context. It does not say whether output is returned as a string or printed to a terminal, nor anything about size limits or truncation of large reports.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with no filler; the verb, resource, and output modes come first. The only inefficiency is the unexplained jargon "FullReport", which consumes words without informing the agent.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a 2-parameter tool with an untyped, undescribed `report` object, no output schema, and no structured parameter documentation, the description is too thin. An agent cannot determine how to obtain or shape the FullReport argument, nor where the rendered output goes, from the definition alone.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must carry parameter meaning, and it only partly does. It enumerates the three formats that the `format` enum already lists (no added meaning, no default), and it never explains the untyped `report` parameter beyond naming "FullReport" โ€” its shape, source, and whether it is required or optional are all left opaque.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

"Render a FullReport as JUnit XML, JSON, or human-readable terminal output" states a specific verb (Render) and resource (FullReport) plus the three output modes, which is clearly distinct from the run* siblings that produce results. It falls short of 5 because it never positions itself relative to those siblings (e.g. that it renders output from runFullSuite), and "FullReport" is an unexplained type name.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no when-to-use guidance at all: nothing says whether this is meant to be called after runFullSuite, on an arbitrary report object, or as a standalone formatter. No alternatives or exclusions are named among the eleven sibling run*/assert*/compare* tools, so the agent must guess the call ordering.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

runAnnotationsAuditC
Read-onlyIdempotent

Apply the heuristic annotation rules (destructive name regex, readOnlyHint conflicts) to every advertised tool.

NameRequiredDescriptionDefault
targetYes
specVersionNo2025-06-18

TDQS

C2.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint=true, idempotentHint=true, destructiveHint=false and openWorldHint=false, so the safety profile is covered. The description adds genuinely new behavioral content by naming the two heuristic rule families it evaluates. However, it does not say whether violations cause a failure, what severity model applies, or what is returned, which a reader of an audit tool needs.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with no filler, and the scope constraint ('every advertised tool') is stated early. It is arguably too terse for the schema complexity it fronts, but nothing is wasted.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, no parameter descriptions, and a required polymorphic target, the description should explain the connection mode and the result shape. It covers only the rule content, leaving an agent unable to construct the target argument from the definition alone.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the required 'target' parameter is a oneOf union of a stdio command spec and an HTTP URL spec with no documented fields. The description says nothing about how to supply a target (command/args/env vs url/headers) or what specVersion changes, so it fails to compensate for a complex, undocumented required parameter.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description states a specific action ('apply the heuristic annotation rules') and target scope ('every advertised tool'), and names the two checks it performs (destructive name regex, readOnlyHint conflicts). It is clear what the tool does, though it never explicitly says the 'target' must be an MCP server connection, nor does it differentiate itself from sibling checks like runToolSchemaValidation.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to run this versus the many sibling audits (runToolSchemaValidation, runCapabilityIntrospection, runFullSuite). No prerequisites, no ordering advice, and no statement of what happens if violations are found.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

runCapabilityIntrospectionB
Read-onlyIdempotent

Compare the capabilities the target advertises in initialize against what tools/list, resources/list, prompts/list actually return.

NameRequiredDescriptionDefault
targetYes
specVersionNo2025-06-18

TDQS

B3.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false and openWorldHint=false, so the safety profile is covered structurally. The description adds real behavioral content beyond that by enumerating the four protocol surfaces being cross-checked. However, it does not disclose what the tool actually does with the target (spawn a subprocess for stdio, open an HTTP connection) or what happens on a mismatch, and that connection behavior sits in some tension with openWorldHint=false.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

One sentence, front-loaded with the verb and the comparison, with no filler or repetition. It is efficient, though arguably under-sized for a tool whose schema has two distinct target variants.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema and no annotations describing the result, yet the description never says what a report looks like, how mismatches are surfaced, or whether failures are returned as errors or as data. For a tool that connects to an arbitrary target and compares four protocol surfaces across four spec versions, this leaves significant gaps an agent must guess at.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0%, so the description must carry parameter meaning and it does not. It references 'the target' but never explains the stdio-versus-http target shapes, nor does it explain what `specVersion` controls or why the default 2025-06-18 matters for the initialize comparison. An agent must read the raw schema to know how to construct a target at all.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description uses a specific verb ('Compare') and names both sides of the comparison: capabilities advertised in `initialize` versus what `tools/list`, `resources/list` and `prompts/list` actually return. That is enough to distinguish it from siblings like runToolSchemaValidation or runAnnotationsAudit, which validate a single surface rather than cross-checking declared versus actual capability. It stops short of explicitly naming those siblings as alternatives.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is only implied: an agent can infer this is the tool for checking that a server's declared capabilities match reality. There is no statement of prerequisites (that the tool connects to, or spawns, the target), no when-not-to-use guidance, and no pointer to runFullSuite or the narrower sibling checks. Minimum viable.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

runFullSuiteB
Read-onlyIdempotent

Run all (or selected) suites and produce a FullReport with summary + per-suite + per-check status.

NameRequiredDescriptionDefault
oauthNo
suiteNo
targetYes
specVersionYes

TDQS

B3.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive and closed-world, so the safety profile is covered. The description adds the output shape (FullReport with summary and per-suite/per-check status), which is genuinely useful since there is no output schema, but it says nothing about the target being launched (stdio subprocess or HTTP endpoint), timeouts, or cost of a full run.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with no filler. It is efficient, though its brevity is partly the reason key parameter and behavioral context is absent.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with required nested target configuration, an oauth mock/real mode, and a version enum, the description covers only the top-level purpose and a sketch of the return value. An agent would not know what 'target' accepts, what specVersion affects, or when oauth mode should be real, all of which are undocumented elsewhere.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across four parameters, including nested objects (target with stdio/http variants and env/headers, oauth with mock/real mode and credentials) and a specVersion enum. The description contributes nothing beyond 'all (or selected)' loosely pointing at the suite parameter, so it fails to compensate for the coverage gap.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and resource ('Run all (or selected) suites') and states the artifact produced ('FullReport with summary + per-suite + per-check status'). It implicitly distinguishes itself from the single-suite siblings (runTransportSuite, runOauthPkceFlow, etc.) by being the aggregate runner, but never names them or says so explicitly.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The parenthetical '(or selected)' hints that the tool can be narrowed, which suggests when to use it over the individual suite runners, but no explicit when-to-use or when-not-to-use guidance is given. An agent still has to infer whether to call this or generateReport / the per-suite tools.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

runJsonRpcComplianceB
Read-onlyIdempotent

Run JSON-RPC 2.0 compliance suite (parse-error, invalid-request, method-not-found, invalid-params, internal-error) against a target MCP server.

NameRequiredDescriptionDefault
targetYes
specVersionYes

TDQS

B3.2/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive, and non-open-world, so the safety profile is covered. The description adds nothing about the fact that the run will spawn a subprocess or open an HTTP connection to the target, nothing about timeouts or how a failing check is reported, and nothing about whether the suite aborts at the first failure. With annotations carrying the safety baseline, a 3 is appropriate.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with the resource first and the enumerated checks in parentheses; no filler, no restatement of the tool name. Every clause carries information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool whose only parameter is a nested oneOf over process-launch and network target shapes, and which has no output schema, the description should say what a run produces (pass/fail report, structured findings?) and how specVersion changes the suite. Neither is addressed, leaving an agent unable to predict the result or construct a valid target without reverse-engineering the schema.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the description provides no parameter-level meaning at all. It never explains that 'target' is a discriminated union of stdio (cmd/cwd/args/env) and http (url/headers) forms, and it never mentions 'specVersion' or why selecting among the four protocol versions matters for compliance results.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

Names a specific verb ('Run') and resource ('JSON-RPC 2.0 compliance suite') and enumerates the five error classes exercised (parse-error, invalid-request, method-not-found, invalid-params, internal-error), so an agent knows exactly what is checked. It does not, however, distinguish itself from runFullSuite or runTransportSuite, which plausibly overlap.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

Usage is implied by the enumerated error classes: call this when you want to verify a server's JSON-RPC error-handling behavior. There is no explicit when-not guidance, no statement of prerequisites (does the target server need to be already running?), and no routing hint versus runFullSuite, which likely subsumes this check.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

runOauthPkceFlowC
Read-onlyIdempotent

Exercise OAuth 2.1 PKCE S256 flow against the target. Mock-AS by default; real-tenant when --oauth-mode=real.

NameRequiredDescriptionDefault
scopesNo
targetYes
clientIdYes
redirectUriYes
specVersionNo2025-06-18
authorizationServerUrlNo

TDQS

C2.9/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnlyHint, idempotentHint, destructiveHint=false, and openWorldHint=false, so the safety profile is largely covered. The description adds the genuinely useful behavioral fact that the default runs against a mock authorization server rather than a live one. It does not explain what the tool returns (tokens? assertions? pass/fail?), how long a real-tenant run takes, or that "real-tenant" mode implies external network traffic that sits awkwardly with openWorldHint=false.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two compact sentences, front-loaded with the operation and followed by the default-mode caveat; nothing is padded. The only cost is that the second sentence's flag syntax is unactionable rather than merely terse.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema, no parameter descriptions, and a real-tenant mode mentioned but not addressable through any argument, an agent cannot reliably invoke this tool or predict its effects. For a 6-parameter, nested-object tool whose mode selection is central to safety, the description leaves too much unspecified.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across 6 parameters, including a nested target object and a specVersion enum, so the description carries the full explanatory burden and supplies almost none of it: no meaning for target, clientId, redirectUri, scopes, or authorizationServerUrl. Worse, it introduces "--oauth-mode=real", a value that has no corresponding parameter, which can send an agent looking for a knob that isn't there.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

"Exercise OAuth 2.1 PKCE S256 flow against the target" gives a specific verb (exercise/run a flow), a specific protocol and code-challenge method (OAuth 2.1 PKCE S256), and a specific target. That distinguishes it well from siblings like runTransportSuite or runJsonRpcCompliance, though it never explicitly contrasts itself with them.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

It states a default ("Mock-AS by default") and an alternative ("real-tenant when --oauth-mode=real"), which is usage-shaped guidance. But the switch is expressed as a CLI flag that does not exist anywhere in the input schema, so an agent has no actionable way to select the mode, and there is no guidance on when this suite should be run instead of the other run* suites.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

runRoundtripSmokeB
Read-onlyIdempotent

Perform one tools/call per advertised tool. Fails on JSON-RPC transport errors, warns on tool-level isError=true.

NameRequiredDescriptionDefault
targetYes
sampleArgsNo
specVersionNo2025-06-18

TDQS

B3.4/5.0
Behavior4/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Goes beyond the annotations by defining the result semantics: transport-level JSON-RPC errors are hard failures, while tool-level isError=true results only warn. That distinction is the key behavioral fact an agent needs. It does not say whether the run spawns processes or opens network connections to the target, nor what the summary output looks like.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two short sentences, front-loaded with the action and then the pass/fail rule. No filler or repetition; every clause carries information.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

For a tool with a nested polymorphic target, an undocumented free-form sampleArgs parameter, and no output schema, the description is too thin. It never explains what 'advertised tool' means (the target server's tools/list output), how sampleArgs maps to those tools, or what the run returns on success versus warning.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across three parameters, including a nested oneOf target (stdio command vs HTTP URL) and a free-form sampleArgs map. The description says nothing about target, sampleArgs, or specVersion, so the agent must infer that sampleArgs supplies arguments for the smoke calls and that specVersion selects protocol behavior entirely from the schema shape.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

States a specific verb and resource: issue exactly one tools/call per advertised tool on a target server. That distinguishes it from runFullSuite and runTransportSuite by scope, though the description never names those siblings to make the contrast explicit.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

'One tools/call per advertised tool' implies a shallow, quick connectivity check, which is implied guidance for choosing it over the fuller runFullSuite. There is no explicit when-to-use, no prerequisite (e.g. server must already be reachable), and no stated exclusion versus runTransportSuite or runJsonRpcCompliance.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

runSpecVersionAssertionB
Read-onlyIdempotent

Drive an initialize handshake and assert the target advertises the expected MCP spec version.

NameRequiredDescriptionDefault
targetYes
expectedVersionYes

TDQS

B3/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly=true, idempotent=true, destructive=false and openWorld=false, so the safety profile is covered. The description adds one genuinely useful behavioral fact: it initiates a live handshake, i.e. it will spawn a stdio process or open an HTTP connection to the target. It does not disclose what happens on a version mismatch (throw, return a report, timeout behavior).

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness5/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with the action and its assertion outcome stated up front, and zero filler. Appropriately sized for a two-parameter assertion tool.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

There is no output schema, so the description should carry the result semantics, but it never says whether success is signaled by a returned value, an exception on mismatch, or a structured diff. Combined with the undescribed dual-mode target parameter, the definition leaves too much for the agent to guess.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the target parameter is a non-trivial oneOf between a stdio object (cmd, args, cwd, env) and an HTTP object (url, headers). The description alludes to 'target' and 'expected version' but never explains that both a local process and a remote URL are accepted, so it fails to compensate for the schema's own lack of descriptions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description pairs a specific verb with a specific resource: it drives an initialize handshake and asserts the advertised MCP spec version. That clearly separates it from siblings such as runCapabilityIntrospection or runFullSuite. It stops short of naming a sibling boundary explicitly, so a 4 rather than a 5.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no stated guidance on when to reach for this narrow version check versus runFullSuite, runCapabilityIntrospection, or the other suite runners, and no preconditions or exclusions. The agent must infer its place in the test suite from the name alone.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

runToolSchemaValidationB
Read-onlyIdempotent

Validate that every advertised tool has a well-formed JSON-Schema inputSchema. Optionally diff against an expected manifest.

NameRequiredDescriptionDefault
targetYes
specVersionNo2025-06-18
expectedManifestNo

TDQS

B3.1/5.0
Behavior3/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive and closed-world, so safety is covered structurally. The description adds that validation operates over 'every advertised tool', but says nothing about how a target server is contacted, what happens on malformed schemas (fail vs report), or whether failures abort the run.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

Two sentences, front-loaded with the primary action and no filler. The trailing 'Optionally diff against an expected manifest' is slightly vague but still earns its place by flagging the second mode.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

This tool has a deeply nested, polymorphic target parameter, no output schema, and no parameter descriptions, so the description carries a heavy burden it does not meet. An agent knows what the tool checks but not how to construct a target, what specVersion affects, or what the result contains.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% and the schema is nested, with a oneOf between a stdio target (cmd/args/cwd/env) and an http target (url/headers), plus specVersion. The description never explains what 'target' accepts or that two transport shapes exist, only naming the optional manifest diff, so it leaves the most complex parameter entirely to the raw schema.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description names a specific verb and object ('Validate that every advertised tool has a well-formed JSON-Schema inputSchema'), which is concrete and actionable. It does not differentiate itself from sibling compareManifests or runAnnotationsAudit, even though the optional manifest diff overlaps with compareManifests, so an agent can't fully disambiguate the pair from the text alone.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines3/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

The word 'Optionally' signals that expectedManifest is a conditional mode, which implicitly tells the agent when to supply it. However, there is no explicit statement of when to use this tool versus compareManifests, runAnnotationsAudit, or runFullSuite, and no prerequisites or exclusions are given.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

runTransportSuiteC
Read-onlyIdempotent

Verify the target's transport layer (stdio line-delimited JSON or Streamable HTTP).

NameRequiredDescriptionDefault
targetYes
transportNoboth
specVersionNo2025-06-18

TDQS

C2.7/5.0
Behavior2/5

Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

Annotations already declare readOnly, idempotent, non-destructive and closed-world, so the safety profile is supplied. The description adds nothing beyond that: it does not say what 'verify' entails, whether subprocesses or network connections are actually made, what a failure looks like, or whether the check is passive or involves repeated probing. Notably, a tool that can spawn an arbitrary command or hit an arbitrary URL is arguably more open-world than openWorldHint=false implies, and the description does nothing to clarify that boundary.

Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

Conciseness4/5

Is the description appropriately sized, front-loaded, and free of redundancy?

A single front-loaded sentence with no filler and no repetition. It is efficient, though its brevity is partly the source of the gaps above rather than a virtue in itself.

Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

Completeness2/5

Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

With no output schema and a nested oneOf target (stdio cmd/args/env/cwd versus http url/headers) plus an undated specVersion enum, the description is far too thin to call this tool correctly. An agent cannot tell what result to expect, how the target shape pairs with the transport filter, or which spec version matters.

Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

Parameters2/5

Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

Schema description coverage is 0% across three parameters, so the description must compensate and does not. It gestures at the two transport kinds via the parenthetical, but never explains the 'transport' param's 'both' default, the interaction between target.kind and transport, or what 'specVersion' selects among the four dated protocol versions.

Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

Purpose4/5

Does the description clearly state what the tool does and how it differs from similar tools?

The description gives a specific verb ('Verify') and resource ('the target's transport layer') and names the two transport modes covered, so an agent knows what is being probed. It does not differentiate itself from close siblings such as runJsonRpcCompliance or runRoundtripSmoke, which also exercise a live target, leaving scope overlap unresolved.

Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

Usage Guidelines2/5

Does the description explain when to use this tool, when not to, or what alternatives exist?

There is no guidance on when to prefer this over the ten sibling run* checks, nor on prerequisites (does the target need to be reachable, does it spawn a subprocess?). The description also does not explain when to choose this single transport check versus runFullSuite, which presumably bundles it.

Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

Tool Schema Changelog

Recent tool additions, removals, and schema changes observed during successful MCP inspections.

  1. 12 tool updates
    • ChangedassertNoBreakingChanges1 field changed
      • removedInput schema / additionalProperties
        Removed value: -false
    • ChangedcompareManifests9 fields changed
      • removedInput schema / additionalProperties
        Removed value: -false
      • removedInput schema / properties / actual / additionalProperties
        Removed value: -false
      • removedInput schema / properties / actual / properties / tools / items / additionalProperties
        Removed value: -false
      • addedInput schema / properties / actual / properties / tools / items / properties / _meta / propertyNames
        Added value: +{
        +  "type": "string"
        +}
      • removedInput schema / properties / actual / properties / tools / items / properties / annotations / additionalProperties
        Removed value: -false
      • removedInput schema / properties / expected / additionalProperties
        Removed value: -false
      • removedInput schema / properties / expected / properties / tools / items / additionalProperties
        Removed value: -false
      • addedInput schema / properties / expected / properties / tools / items / properties / _meta / propertyNames
        Added value: +{
        +  "type": "string"
        +}
      • removedInput schema / properties / expected / properties / tools / items / properties / annotations / additionalProperties
        Removed value: -false
    • ChangedgenerateReport1 field changed
      • removedInput schema / additionalProperties
        Removed value: -false
    • ChangedrunAnnotationsAudit3 fields changed
      • removedInput schema / additionalProperties
        Removed value: -false
      • removedInput schema / properties / target / anyOf
        Removed value: -[
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "args": {
        -        "items": {
        -          "type": "string"
        -        },
        -        "type": "array"
        -      },
        -      "cmd": {
        -        "minLength": 1,
        -        "type": "string"
        -      },
        -      "cwd": {
        -        "type": "string"
        -      },
        -      "env": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "stdio",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "cmd"
        -    ],
        -    "type": "object"
        -  },
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "headers": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "http",
        -        "type": "string"
        -      },
        -      "url": {
        -        "format": "uri",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "url"
        -    ],
        -    "type": "object"
        -  }
        -]
      • addedInput schema / properties / target / oneOf
        Added value: +[
        +  {
        +    "properties": {
        +      "args": {
        +        "items": {
        +          "type": "string"
        +        },
        +        "type": "array"
        +      },
        +      "cmd": {
        +        "minLength": 1,
        +        "type": "string"
        +      },
        +      "cwd": {
        +        "type": "string"
        +      },
        +      "env": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "stdio",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "cmd"
        +    ],
        +    "type": "object"
        +  },
        +  {
        +    "properties": {
        +      "headers": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "http",
        +        "type": "string"
        +      },
        +      "url": {
        +        "format": "uri",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "url"
        +    ],
        +    "type": "object"
        +  }
        +]
    • ChangedrunCapabilityIntrospection3 fields changed
      • removedInput schema / additionalProperties
        Removed value: -false
      • removedInput schema / properties / target / anyOf
        Removed value: -[
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "args": {
        -        "items": {
        -          "type": "string"
        -        },
        -        "type": "array"
        -      },
        -      "cmd": {
        -        "minLength": 1,
        -        "type": "string"
        -      },
        -      "cwd": {
        -        "type": "string"
        -      },
        -      "env": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "stdio",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "cmd"
        -    ],
        -    "type": "object"
        -  },
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "headers": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "http",
        -        "type": "string"
        -      },
        -      "url": {
        -        "format": "uri",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "url"
        -    ],
        -    "type": "object"
        -  }
        -]
      • addedInput schema / properties / target / oneOf
        Added value: +[
        +  {
        +    "properties": {
        +      "args": {
        +        "items": {
        +          "type": "string"
        +        },
        +        "type": "array"
        +      },
        +      "cmd": {
        +        "minLength": 1,
        +        "type": "string"
        +      },
        +      "cwd": {
        +        "type": "string"
        +      },
        +      "env": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "stdio",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "cmd"
        +    ],
        +    "type": "object"
        +  },
        +  {
        +    "properties": {
        +      "headers": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "http",
        +        "type": "string"
        +      },
        +      "url": {
        +        "format": "uri",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "url"
        +    ],
        +    "type": "object"
        +  }
        +]
    • ChangedrunFullSuite4 fields changed
      • removedInput schema / additionalProperties
        Removed value: -false
      • removedInput schema / properties / oauth / additionalProperties
        Removed value: -false
      • removedInput schema / properties / target / anyOf
        Removed value: -[
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "args": {
        -        "items": {
        -          "type": "string"
        -        },
        -        "type": "array"
        -      },
        -      "cmd": {
        -        "minLength": 1,
        -        "type": "string"
        -      },
        -      "cwd": {
        -        "type": "string"
        -      },
        -      "env": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "stdio",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "cmd"
        -    ],
        -    "type": "object"
        -  },
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "headers": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "http",
        -        "type": "string"
        -      },
        -      "url": {
        -        "format": "uri",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "url"
        -    ],
        -    "type": "object"
        -  }
        -]
      • addedInput schema / properties / target / oneOf
        Added value: +[
        +  {
        +    "properties": {
        +      "args": {
        +        "items": {
        +          "type": "string"
        +        },
        +        "type": "array"
        +      },
        +      "cmd": {
        +        "minLength": 1,
        +        "type": "string"
        +      },
        +      "cwd": {
        +        "type": "string"
        +      },
        +      "env": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "stdio",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "cmd"
        +    ],
        +    "type": "object"
        +  },
        +  {
        +    "properties": {
        +      "headers": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "http",
        +        "type": "string"
        +      },
        +      "url": {
        +        "format": "uri",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "url"
        +    ],
        +    "type": "object"
        +  }
        +]
    • ChangedrunJsonRpcCompliance3 fields changed
      • removedInput schema / additionalProperties
        Removed value: -false
      • removedInput schema / properties / target / anyOf
        Removed value: -[
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "args": {
        -        "items": {
        -          "type": "string"
        -        },
        -        "type": "array"
        -      },
        -      "cmd": {
        -        "minLength": 1,
        -        "type": "string"
        -      },
        -      "cwd": {
        -        "type": "string"
        -      },
        -      "env": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "stdio",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "cmd"
        -    ],
        -    "type": "object"
        -  },
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "headers": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "http",
        -        "type": "string"
        -      },
        -      "url": {
        -        "format": "uri",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "url"
        -    ],
        -    "type": "object"
        -  }
        -]
      • addedInput schema / properties / target / oneOf
        Added value: +[
        +  {
        +    "properties": {
        +      "args": {
        +        "items": {
        +          "type": "string"
        +        },
        +        "type": "array"
        +      },
        +      "cmd": {
        +        "minLength": 1,
        +        "type": "string"
        +      },
        +      "cwd": {
        +        "type": "string"
        +      },
        +      "env": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "stdio",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "cmd"
        +    ],
        +    "type": "object"
        +  },
        +  {
        +    "properties": {
        +      "headers": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "http",
        +        "type": "string"
        +      },
        +      "url": {
        +        "format": "uri",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "url"
        +    ],
        +    "type": "object"
        +  }
        +]
    • ChangedrunOauthPkceFlow3 fields changed
      • removedInput schema / additionalProperties
        Removed value: -false
      • removedInput schema / properties / target / additionalProperties
        Removed value: -false
      • addedInput schema / properties / target / properties / headers / propertyNames
        Added value: +{
        +  "type": "string"
        +}
    • ChangedrunRoundtripSmoke4 fields changed
      • removedInput schema / additionalProperties
        Removed value: -false
      • addedInput schema / properties / sampleArgs / propertyNames
        Added value: +{
        +  "type": "string"
        +}
      • removedInput schema / properties / target / anyOf
        Removed value: -[
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "args": {
        -        "items": {
        -          "type": "string"
        -        },
        -        "type": "array"
        -      },
        -      "cmd": {
        -        "minLength": 1,
        -        "type": "string"
        -      },
        -      "cwd": {
        -        "type": "string"
        -      },
        -      "env": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "stdio",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "cmd"
        -    ],
        -    "type": "object"
        -  },
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "headers": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "http",
        -        "type": "string"
        -      },
        -      "url": {
        -        "format": "uri",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "url"
        -    ],
        -    "type": "object"
        -  }
        -]
      • addedInput schema / properties / target / oneOf
        Added value: +[
        +  {
        +    "properties": {
        +      "args": {
        +        "items": {
        +          "type": "string"
        +        },
        +        "type": "array"
        +      },
        +      "cmd": {
        +        "minLength": 1,
        +        "type": "string"
        +      },
        +      "cwd": {
        +        "type": "string"
        +      },
        +      "env": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "stdio",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "cmd"
        +    ],
        +    "type": "object"
        +  },
        +  {
        +    "properties": {
        +      "headers": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "http",
        +        "type": "string"
        +      },
        +      "url": {
        +        "format": "uri",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "url"
        +    ],
        +    "type": "object"
        +  }
        +]
    • ChangedrunSpecVersionAssertion3 fields changed
      • removedInput schema / additionalProperties
        Removed value: -false
      • removedInput schema / properties / target / anyOf
        Removed value: -[
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "args": {
        -        "items": {
        -          "type": "string"
        -        },
        -        "type": "array"
        -      },
        -      "cmd": {
        -        "minLength": 1,
        -        "type": "string"
        -      },
        -      "cwd": {
        -        "type": "string"
        -      },
        -      "env": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "stdio",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "cmd"
        -    ],
        -    "type": "object"
        -  },
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "headers": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "http",
        -        "type": "string"
        -      },
        -      "url": {
        -        "format": "uri",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "url"
        -    ],
        -    "type": "object"
        -  }
        -]
      • addedInput schema / properties / target / oneOf
        Added value: +[
        +  {
        +    "properties": {
        +      "args": {
        +        "items": {
        +          "type": "string"
        +        },
        +        "type": "array"
        +      },
        +      "cmd": {
        +        "minLength": 1,
        +        "type": "string"
        +      },
        +      "cwd": {
        +        "type": "string"
        +      },
        +      "env": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "stdio",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "cmd"
        +    ],
        +    "type": "object"
        +  },
        +  {
        +    "properties": {
        +      "headers": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "http",
        +        "type": "string"
        +      },
        +      "url": {
        +        "format": "uri",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "url"
        +    ],
        +    "type": "object"
        +  }
        +]
    • ChangedrunToolSchemaValidation7 fields changed
      • removedInput schema / additionalProperties
        Removed value: -false
      • removedInput schema / properties / expectedManifest / additionalProperties
        Removed value: -false
      • removedInput schema / properties / expectedManifest / properties / tools / items / additionalProperties
        Removed value: -false
      • addedInput schema / properties / expectedManifest / properties / tools / items / properties / _meta / propertyNames
        Added value: +{
        +  "type": "string"
        +}
      • removedInput schema / properties / expectedManifest / properties / tools / items / properties / annotations / additionalProperties
        Removed value: -false
      • removedInput schema / properties / target / anyOf
        Removed value: -[
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "args": {
        -        "items": {
        -          "type": "string"
        -        },
        -        "type": "array"
        -      },
        -      "cmd": {
        -        "minLength": 1,
        -        "type": "string"
        -      },
        -      "cwd": {
        -        "type": "string"
        -      },
        -      "env": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "stdio",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "cmd"
        -    ],
        -    "type": "object"
        -  },
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "headers": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "http",
        -        "type": "string"
        -      },
        -      "url": {
        -        "format": "uri",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "url"
        -    ],
        -    "type": "object"
        -  }
        -]
      • addedInput schema / properties / target / oneOf
        Added value: +[
        +  {
        +    "properties": {
        +      "args": {
        +        "items": {
        +          "type": "string"
        +        },
        +        "type": "array"
        +      },
        +      "cmd": {
        +        "minLength": 1,
        +        "type": "string"
        +      },
        +      "cwd": {
        +        "type": "string"
        +      },
        +      "env": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "stdio",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "cmd"
        +    ],
        +    "type": "object"
        +  },
        +  {
        +    "properties": {
        +      "headers": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "http",
        +        "type": "string"
        +      },
        +      "url": {
        +        "format": "uri",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "url"
        +    ],
        +    "type": "object"
        +  }
        +]
    • ChangedrunTransportSuite3 fields changed
      • removedInput schema / additionalProperties
        Removed value: -false
      • removedInput schema / properties / target / anyOf
        Removed value: -[
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "args": {
        -        "items": {
        -          "type": "string"
        -        },
        -        "type": "array"
        -      },
        -      "cmd": {
        -        "minLength": 1,
        -        "type": "string"
        -      },
        -      "cwd": {
        -        "type": "string"
        -      },
        -      "env": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "stdio",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "cmd"
        -    ],
        -    "type": "object"
        -  },
        -  {
        -    "additionalProperties": false,
        -    "properties": {
        -      "headers": {
        -        "additionalProperties": {
        -          "type": "string"
        -        },
        -        "type": "object"
        -      },
        -      "kind": {
        -        "const": "http",
        -        "type": "string"
        -      },
        -      "url": {
        -        "format": "uri",
        -        "type": "string"
        -      }
        -    },
        -    "required": [
        -      "kind",
        -      "url"
        -    ],
        -    "type": "object"
        -  }
        -]
      • addedInput schema / properties / target / oneOf
        Added value: +[
        +  {
        +    "properties": {
        +      "args": {
        +        "items": {
        +          "type": "string"
        +        },
        +        "type": "array"
        +      },
        +      "cmd": {
        +        "minLength": 1,
        +        "type": "string"
        +      },
        +      "cwd": {
        +        "type": "string"
        +      },
        +      "env": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "stdio",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "cmd"
        +    ],
        +    "type": "object"
        +  },
        +  {
        +    "properties": {
        +      "headers": {
        +        "additionalProperties": {
        +          "type": "string"
        +        },
        +        "propertyNames": {
        +          "type": "string"
        +        },
        +        "type": "object"
        +      },
        +      "kind": {
        +        "const": "http",
        +        "type": "string"
        +      },
        +      "url": {
        +        "format": "uri",
        +        "type": "string"
        +      }
        +    },
        +    "required": [
        +      "kind",
        +      "url"
        +    ],
        +    "type": "object"
        +  }
        +]
  2. 12 tool updatesv0.2.1
    • First observedassertNoBreakingChanges
    • First observedcompareManifests
    • First observedgenerateReport
    • First observedrunAnnotationsAudit
    • First observedrunCapabilityIntrospection
    • First observedrunFullSuite
    • First observedrunJsonRpcCompliance
    • First observedrunOauthPkceFlow
    • First observedrunRoundtripSmoke
    • First observedrunSpecVersionAssertion
    • First observedrunToolSchemaValidation
    • First observedrunTransportSuite

TDQS

B3.4/5.0

Scored across 12 tools

Disambiguation5/5

Each tool targets a distinct conformance concern: JSON-RPC framing, spec version, transport, OAuth, schema shape, capability consistency, roundtrip, annotations, manifest diffing, and report rendering. runFullSuite intentionally aggregates the others but is clearly described as an orchestrator rather than a duplicate check. No two tools appear to do the same thing.

Naming Consistency4/5

Nine tools share the runXSuite/runXAssertion camelCase verb_noun pattern, and the remaining three use equally descriptive distinct verbs (generateReport, compareManifests, assertNoBreakingChanges) that signal post-processing rather than test execution. The convention is predictable, though the run-prefix is applied to a few tools that are checks rather than suites.

Tool Count5/5

Twelve tools is well within the ideal range and each maps to a concrete, non-redundant conformance check or reporting step. Nothing feels padded, and the granular suite tools plus a full-suite orchestrator is a natural decomposition.

Completeness4/5

The surface covers the major conformance axes: JSON-RPC, transport, spec version, OAuth PKCE, tool schemas, capability introspection, roundtrips, annotations, reporting, and regression detection. Gaps remain for server-initiated features (sampling, roots, elicitation), notification/progress behavior, and non-PKCE auth methods, but these are extensions rather than dead ends.

Maintenance

ActivityActive
ResponsivenessNo recent community issues

Related MCP Connectors

Related MCP Servers

  • A
    license
    Not graded
    quality
    D
    maintenance
    A TypeScript-based test implementation for the Model Context Protocol (MCP) with example configurations and Jest test cases. Provides a development foundation for building and testing MCP server functionality.
    3 npm
    MIT
  • A
    license
    Not graded
    quality
    D
    maintenance
    A comprehensive reference implementation demonstrating all features of the Model Context Protocol (MCP) specification, serving as documentation, learning resource, and testing tool for MCP implementations.
    1
    MIT
  • A
    license
    Not graded
    quality
    D
    maintenance
    A comprehensive testing server for validating MCP scanner tools and implementations, providing diverse tools, resources, and prompts that exercise various aspects of the Model Context Protocol specification.
    8 npm
    MIT