Server Quality Checklist

Tool Scores

• generate_specB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions discovering capabilities but omits critical details: that it will connect to the server, potentially execute calls (especially with depth='deep'), and that it returns a YAML spec. The actual behavior of initiating network connections and running discovery actions is not transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no wasted words. The action and result are front-loaded. Every word adds value, and the structure is clear and direct.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 9 parameters and no output schema, the description is too terse. It does not explain the discovery mechanism, how transport selection affects required parameters, or how the output spec is structured. An agent lacks the operational context needed to invoke the tool correctly, especially for complex scenarios like specifying headers or depth.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description does not add parameter-specific semantics beyond what the schema already provides. It does not elaborate on how parameters like depth or include/exclude affect the generated spec.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states the tool generates a YAML test spec by discovering MCP server capabilities and returns a ready-to-run spec. The verb 'generate' is specific, and the resource is clearly a test spec. It distinguishes from sibling tools (explain_failure, inspect_server, run_spec) which serve different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. The description does not mention prerequisites, when not to use it, or how it compares to siblings like inspect_server or run_spec. Agents must infer usage context from the name and schema alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspect_serverB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It implies a read-only inspection but does not disclose connection failure behaviors, timeouts, or side effects. For a tool that connects to an external server, this is insufficient transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no unnecessary words. Every part serves to define the tool's core function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description lacks information about return values (no output schema) and does not cover what happens after connection (e.g., what 'inspect' returns). For a tool with 6 parameters and no annotations, this is incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents parameter meanings. The description adds no extra context beyond the schema, meeting the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses specific verbs and nouns: 'connect to an MCP server and inspect its capabilities, version, and supported features.' It clearly distinguishes from siblings like explain_failure (explain), generate_spec (generate), and run_spec (run), making the tool's purpose unique.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no explicit guidance on when to use this tool versus alternatives. It only states its function, leaving the agent to infer from sibling names. No 'when to use' or 'when not to use' conditions are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_specA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It mentions 'return results' but does not specify output format, side effects, or safety profile. Important behavioral details are missing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two efficient sentences: first states purpose, second clarifies parameter usage. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations or output schema, the description is minimal but covers the core purpose and parameter constraints. It lacks broader context like expected output, error handling, or integration with sibling tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description adds value by clarifying mutual exclusivity ('either specText...or specPath') and the requirement ('At least one is required'), which is not in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Run a YAML test spec against an MCP server and return results,' providing a specific verb and resource. It distinguishes from sibling tools (explain_failure, generate_spec, inspect_server) by focusing on execution.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description notes that at least one of specText or specPath is required, but lacks explicit guidance on when to use this tool versus siblings or when not to use it. The context is implied by the tool's name and sibling differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_failureA
  3.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description bears full responsibility for behavioral disclosure. It mentions heuristic classification and recommendations but does not describe any side effects, authorization needs, rate limits, or what happens with invalid input. The brief description leaves significant behavioral ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence that conveys the purpose and input requirement without redundancy. Every word serves a purpose, making it highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While the input schema is fully described, the tool has no output schema, and the description lacks details about the output format or structure. The mention of 'heuristic classification and actionable recommendations' is vague, leaving the agent uncertain about what the tool returns.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, detailing all nested properties. The description adds meaning by specifying that the runResult parameter should be the structured result from run_spec, which contextualizes the parameter beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool analyzes test run results to explain failures with heuristic classification and actionable recommendations. It specifies the input source as 'the structured result from run_spec,' which differentiates it from sibling tools like generate_spec, inspect_server, and run_spec.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly guides usage by requiring the structured result from run_spec, indicating it should be used after a run_spec call. However, it does not explicitly state when not to use it or provide alternative tools for similar purposes.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.