Server Quality Checklist

Tool Scores

• qa_get_reportB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden but only states the basic action. It does not disclose consequences of missing reports, required permissions, side effects, or error behavior. This leaves the agent with incomplete understanding.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very concise—two sentences that cover the core functionality and an optional parameter. No wasted words, and the structure is front-loaded with the primary action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is adequate for a simple retrieval tool with well-documented parameters, but it lacks details about the output format (no output schema) and error conditions. It does not explain what 'last test report' means or how failures are handled, leaving some gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already explains both parameters. The description adds slight value by noting that the format parameter can override the config default, but it does not elaborate on other parameter details or add meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool retrieves the last test report for a project, using a specific verb ('Retrieve') and resource ('test report'). It distinguishes itself from sibling tools like qa_run_tests (which runs tests) and qa_check_config (which checks config).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide any guidance on when to use this tool versus alternatives. It mentions an optional format parameter but lacks explicit context about when the tool is appropriate or what prerequisites exist.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• qa_check_configA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose all behavioral traits. It states the tool returns validation results with error messages and line numbers, but does not indicate whether it modifies any files, requires permissions, or has side effects. While not contradictory, it lacks depth for a validation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no wasted words. It front-loads the core purpose and then specifies the output format, making it efficient for an AI agent to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (single parameter, no output schema, no annotations), the description is reasonably complete. It explains what it validates and what results to expect, though it could mention that it is a read-only operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for the single parameter 'projectPath', and its schema description adequately explains it. The tool description does not add any additional meaning beyond the schema, warranting a baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool validates a specific file (.mcp-qa-config.json) and returns validation results. The verb 'Validate' and resource 'project's .mcp-qa-config.json file' are specific, and it distinguishes itself from sibling tools (e.g., qa_run_tests, qa_list_modules) by focusing on configuration validation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives (e.g., qa_run_module, qa_run_tests) or context for usage. It does not mention prerequisites, when validation is needed, or when it might not be appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• qa_run_testsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses loading config, executing modules, and returning a report. However, it does not mention safety (e.g., whether tests can modify state), idempotency, or error behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no fluff. Front-loaded with the core action and quickly provides additional details. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, the description mentions returning a 'formatted report' but lacks specifics on the report format or contents. Also lacks error handling or edge case info. Adequate but not comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds slight context (e.g., 'Only run tests whose names contain this substring' for filter) but largely restates schema descriptions. No deep semantic enrichment.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Run QA tests for a project' with a specific verb and resource, and references the config file .mcp-qa-config.json. It distinguishes from siblings like qa_run_module by mentioning optional filtering by modules or test name.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage (when you want to run tests) but does not explicitly state when to use this tool versus alternatives like qa_run_module or qa_list_modules. No exclusions or 'when not to use' guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• qa_run_moduleA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden for behavioral disclosure. It does not reveal what happens on module failure, whether it modifies state, access permissions needed, or output format. For a test-running tool, basic behavioral info is missing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no unnecessary words. It front-loads the action and purpose immediately, making it easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with three simple parameters and no output schema, the description covers the main purpose and usage scenario. It mentions 'targeted testing' and gives examples. It could mention the requirement of .mcp-qa-config.json (implied in projectPath schema), but overall it is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema coverage is 100%, so the schema already documents all three parameters adequately. The description adds no additional meaning or context beyond what the schema provides, adhering to the baseline of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool runs a single QA test module for a project, using a specific verb ('Run') and resource ('single QA test module'). It also distinguishes from siblings like qa_run_tests by emphasizing targeted testing of a specific area (e.g., 'just API tests or just VM checks').

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit context, recommending use for targeted testing of a specific area (e.g., 'just API tests or just VM checks'). It implies when to use this tool over alternatives like qa_run_tests, but does not explicitly state when not to use it or name alternatives directly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• qa_list_modulesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It discloses the main behavior (listing modules) and conditional behavior (with projectPath), but does not mention read-only nature, authentication needs, or rate limits. For a simple list tool, this is adequate but not exceptional.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the main purpose, and uses minimal words to convey the tool's function and conditional behavior. No wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity of the tool (one optional parameter, no output schema), the description covers the essential functionality. It could mention return format or that it returns a list of module names and statuses, but for a list tool, it is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (one parameter fully described). The tool description repeats the same information about the optional parameter, adding no new meaning beyond what the schema already provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and the resource 'QA testing modules', and distinguishes itself from sibling tools like qa_run_module or qa_check_config by focusing on listing rather than execution or configuration.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use the optional projectPath parameter (to see enabled modules for a specific project), which provides clear usage guidance. However, it does not explicitly state when not to use alternatives, but the sibling tools are sufficiently different.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.