Server Quality Checklist

Tool Scores

• audit_supply_chainB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the tool returns a trust score (0-100) but lacks details on rate limits, authentication needs, error handling, or data sources. For a tool that likely queries external APIs, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that efficiently conveys the tool's purpose, key metrics, and output without any wasted words. It is appropriately sized and front-loaded with essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (analyzing npm packages) and lack of annotations or output schema, the description is partially complete. It covers the purpose and output type but misses behavioral details like data freshness, limitations, or error cases, leaving gaps for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters (package_name and version). The description implies these parameters but adds no syntax, format, or contextual details beyond what the schema provides, meeting the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('audit'), resource ('npm package's supply chain health'), and scope ('publish date, weekly downloads, maintainer count, dependency count, and known CVEs'), distinguishing it from sibling tools like check_prompt_injection or scan_directory which target different resources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like scan_package (a sibling tool), nor are there any prerequisites or exclusions mentioned. The description only states what it does, not when it's appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_prompt_injectionA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It describes what the tool does (scanning for specific threats) but lacks details on behavioral traits such as performance characteristics (e.g., speed, accuracy), error handling, or output format. While it mentions what is scanned, it does not disclose how results are returned or any limitations, leaving gaps in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, dense sentence that efficiently conveys the tool's purpose, target resources, and scanning criteria without unnecessary words. It is front-loaded with the core action ('Scan text') and avoids redundancy, making every part of the sentence contribute directly to understanding the tool's function.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a security scanning tool with no annotations and no output schema, the description is moderately complete. It covers what the tool does and what it scans for, but lacks details on output (e.g., what results look like) and behavioral aspects like error handling or limitations. This is adequate for basic understanding but has clear gaps for effective agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('text' and 'label') with descriptions. The description adds no additional meaning or context beyond what the schema provides, such as examples or constraints on parameter values. This meets the baseline for high schema coverage but does not enhance parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Scan text') and the purpose ('for prompt injection attacks, hidden instructions, zero-width characters, jailbreak patterns, and instruction overrides'), with explicit mention of the target resources ('tool descriptions, tool responses, MCP server manifest content'). It distinguishes itself from siblings like audit_supply_chain or scan_directory by focusing on text-based security scanning rather than supply chain or file system analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by listing target resources ('tool descriptions, tool responses, MCP server manifest content'), suggesting it should be used for scanning such content. However, it does not explicitly state when to use this tool versus alternatives like scan_package or provide exclusions (e.g., when not to use it for non-text data). The guidance is present but not comprehensive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_directoryA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions the scanning action and targets but does not describe what the tool returns (e.g., report format, findings), potential side effects (e.g., file system access, performance impact), or any constraints like permissions needed or rate limits. This leaves significant gaps for a security scanning tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with zero waste: the first sentence specifies the action and targets, and the second provides usage context. It is appropriately sized and front-loaded with the core purpose, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a security scanning tool, no annotations, and no output schema, the description is incomplete. It lacks details on what the scan returns (e.g., findings, errors), behavioral traits like safety or performance, and does not compensate for the absence of structured output information, making it inadequate for full agent understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with the single parameter 'path' well-documented in the schema as 'Absolute or relative path to the MCP server directory'. The description does not add any additional meaning or examples beyond what the schema provides, so it meets the baseline for high coverage without extra value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('scan', 'for') and resources ('local MCP server directory'), listing concrete targets like backdoors, exfiltration code, obfuscation, and dangerous patterns. It distinguishes from siblings by focusing on directory scanning rather than supply chain, prompt injection, or package analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('for MCP servers installed locally or cloned from GitHub'), but it does not explicitly state when not to use it or name alternatives among the sibling tools. The guidance is helpful but lacks explicit exclusions or comparisons.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_packageA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden and does well by disclosing key behaviors: it downloads packages, performs multiple security checks, and returns a structured report with severity ratings. It doesn't mention performance characteristics, rate limits, or authentication needs, but covers the core operational behavior adequately for a scanning tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, dense sentence that efficiently conveys purpose, scope, and output without wasted words. It's front-loaded with the core action and resource, making it immediately understandable while remaining appropriately sized for the tool's complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity, no annotations, and no output schema, the description does well by explaining what the tool does and what it returns. However, it could better address behavioral aspects like error handling or performance expectations to be fully complete for a security scanning operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters thoroughly. The description adds no additional parameter semantics beyond what's in the schema, maintaining the baseline score of 3 for adequate but not enhanced parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Download an npm MCP server package and scan it') and resource ('npm MCP server package'), with detailed scope ('for backdoors, exfiltration code, obfuscation, dangerous code execution, and supply chain risks'). It distinguishes from sibling tools like 'audit_supply_chain' by focusing on package scanning rather than broader supply chain auditing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for npm package security scanning, but provides no explicit guidance on when to use this tool versus alternatives like 'audit_supply_chain' or 'scan_directory'. It doesn't mention prerequisites, exclusions, or comparative scenarios, leaving usage context to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.