Server Quality Checklist

Tool Scores

• scan_tool_poisoningA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses what it scans for and input methods, but lacks details about side effects (e.g., whether source_url clones a repo, output format, or if it is read-only). It adds context but is not fully exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: first states purpose, second covers usage and input options. No wasted words, front-loaded with critical information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite good input guidance, the description omits output information (what returns, format, status codes). With no output schema, this is a significant gap for an agent to understand the tool's behavior fully.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds meaningful context by explaining the structure of tool_definitions objects and the conditional usage of source_url. This goes beyond the schema's descriptions, which only list properties.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses specific verbs and resources: 'Scan MCP tool definitions for hidden instructions, unicode tricks, obfuscated payloads, and manipulation patterns.' It clearly distinguishes from sibling tools that deal with packages or servers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use it ('check if a server's tools contain poisoning indicators') and provides input options (tool_definitions or source_url). However, it does not mention when not to use it or alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• submit_reportA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It states the report becomes publicly available, implying permanence, but does not disclose other behaviors like permissions needed, error cases, or idempotency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two clear sentences: one for what it does, one for when to use it. No redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given one parameter and no output schema, the description covers purpose, prerequisite, and outcome. Could be improved by mentioning success/failure indicators or error handling, but still sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with comprehensive subfield descriptions. The tool description adds no extra meaning beyond 'submit a completed report', so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool submits a completed security audit report to AgentAudit registry, distinguishing it from sibling tools like audit_package (analysis) and scan_tool_poisoning (scanning).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to call after analyzing code from audit_package, providing clear usage context. However, does not explicitly mention when not to use or list alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_packageA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It transparently describes the return values: 'risk score, findings, and official audit status,' and explains behavior for missing packages (suggests running an audit). No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the core purpose. Every sentence earns its place—first states action, second adds usage guidance and fallback behavior. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (1 param, no output schema, no annotations), the description is sufficiently complete. It covers purpose, returns, and usage context. Missing information about authentication or rate limits is acceptable given the tool's nature.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds the example values 'fastmcp', 'mongodb-mcp-server', but this is largely redundant with the schema's description. No additional semantic meaning beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Look up a package in the AgentAudit security registry.' It uses a specific verb and resource, and immediately distinguishes itself from siblings by instructing to use this first for installations or safety queries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use: 'USE THIS FIRST whenever the user wants to install, add, evaluate, or learn about a specific MCP server or package.' It also provides context for not-found cases. However, it lacks explicit exclusions or direct mention of alternative sibling tools like audit_package.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• audit_packageA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Despite no annotations, the description discloses the 3-pass methodology, that it clones the repo, and that the user must analyze and call submit_report. It does not detail side effects like permissions or error states, but is sufficient for understanding behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise, front-loaded with purpose, methodology, and usage guidance. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given one parameter, no output schema, and good sibling context, the description is nearly complete. It could mention the return format explicitly but is sufficient for agent decision-making.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a clear description for source_url. The description adds 'Clones the repo' context, but this is already implied. No additional parameter meaning beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it performs a 'Deep security audit of a Git repository' with a specific methodology (UNDERSTAND → DETECT → CLASSIFY). It distinguishes from sibling tools like check_package (pre-check) and submit_report (post-audit) by outlining the workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs to 'use check_package FIRST to see if an audit already exists — only use this for unaudited packages or when a fresh audit is requested.' This provides clear guidance on when to use and when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• discover_serversA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description discloses the action (scanning local config files, listing servers) and implies it is a read-only, non-destructive operation. Could explicitly state no side effects or permissions required, but current text adequately conveys the behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no waste: first sentence declares action and scope, second sentence provides usage guidance and alternative. Front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity, the description covers purpose and usage fully. No output schema, but the tool's output (list of servers) is implied. Could benefit from mentioning the output format or what each entry contains, but still sufficient for a discovery tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single parameter (check_registry). The description adds no extra meaning beyond the schema's own description, so baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool scans local config files to list already installed MCP servers, specifying the exact apps (Claude Desktop, Cursor, Windsurf, VS Code). It differentiates from sibling check_package by explicitly contrasting use cases.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use ('when user wants to review/list existing servers') and when-not-to-use ('install, evaluate, or look up a specific package'), with a direct alternative ('use check_package instead').

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.