Server Quality Checklist

Tool Scores

• grade_mcp_serverA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It transparently describes the outputs (grade, per-category scores, finding counts, prioritized list) but does not mention potential side effects such as executing local commands or making network calls when grading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is structured into four sentences, front-loaded with the core action, and each sentence adds distinct information: grading scope, graded properties, return format, and usage scenarios. It is dense but avoids redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, what is measured, return format, and when to use it, compensating for the lack of an output schema. Minor gaps like execution side effects exist, but overall it is a well-rounded description for a two-parameter evaluation tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% with detailed explanations of both target and max_findings. The tool description adds no additional parameter-specific meaning beyond the schema, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Scores an MCP server on agent usability (A–F)' which clearly identifies the action and resource. It distinguishes from sibling tools by focusing on grading a server rather than explaining or listing rules.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage context: 'Use this before depending on a third-party server, or after changing your own catalog.' It does not mention exclusions or alternatives, but the primary use case is clearly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_ruleA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden for behavioral disclosure. It clearly states the tool 'returns' information, implying a read-only, non-mutating operation. It also details the content of the rationale, which provides realistic expectations. However, it does not explicitly mention error handling, side effects, or authentication requirements, which would be beneficial for full transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the purpose, and every clause adds value. The first sentence explains what it returns, and the second gives usage context. There is no redundancy or irrelevant detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (one parameter, no output schema, no annotations). The description compensates for the missing output schema by listing the content of the rationale, which suggests the return structure. It also explains the tool's role in the broader workflow. While it could be more explicit about the exact return format, the description is complete enough for an experienced user to understand and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the existing parameter documentation is complete. The tool description does not add additional meaning about the parameter beyond what the schema already provides (e.g., format, case-insensitivity, examples). It refers to 'rule ID' generically but does not elaborate further, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Returns') and identifies a concrete resource ('the full rationale for one grading rule') with explicit content categories ('what it detects, why it degrades agent behaviour, and how to fix it'). It clearly distinguishes itself from siblings by focusing on explaining a single rule rather than listing or grading.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool ('after grade_mcp_server reports a finding you want to understand or dispute') and what not to do ('rather than guessing what a rule ID means'). This provides clear situational guidance and references the sibling tool as the trigger context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_grading_rulesA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries the burden. It discloses the return content (identifier, severity, one-line summary) and implies read-only behavior. It does not cover edge cases or rate limits, but for a simple list query, this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the core function, no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given one parameter and no output schema, the description fully explains the purpose, return contents, and usage context. Minor omission: no mention of output list shape or size, but this is not critical.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a detailed enum description for category, so the description adds no additional semantics. Baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly identifies the action (lists) and resource (every rule mcpgrade applies), and specifies the output fields (identifier, severity, one-line summary). Distinguishes itself from explain_rule by noting it can provide identifiers for that tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states two use cases: to see what is checked before grading, and to find an identifier to pass to explain_rule. This differentiates when to use this vs. explain_rule, though it doesn't mention grade_mcp_server directly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.