Server Quality Checklist

Tool Scores

• scan_tool_poisoningA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It mentions scanning and extraction from code but omits important behaviors such as network access/repo cloning (only mentioned in the schema's source_url parameter description), potential side effects, and what the output format is. This is a significant gap for a tool that may clone external repositories.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the main action, and contains no redundant words. It efficiently conveys the purpose, usage context, and input methods.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the purpose and inputs well, but it lacks information about the output format and edge cases (e.g., what happens if neither tool_definitions nor source_url is provided). With no output schema to compensate, this leaves some gaps, though the core functionality is adequately described.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds value by clarifying the OR relationship between tool_definitions and source_url, which is not explicit in the schema. However, it doesn't add further semantics beyond that, so a 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('scan'), a clear resource ('MCP tool definitions'), and a detailed scope (hidden instructions, unicode tricks, obfuscated payloads, manipulation patterns). This clearly distinguishes the tool from siblings like discover_servers or audit_package.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states 'Use this to check if a server's tools contain poisoning indicators', providing a clear context for when to use it. It also explains the two input modes (tool_definitions or source_url). It doesn't mention exclusions or alternatives, but the sibling tools serve different purposes, so the context is sufficient.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• audit_packageA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that it clones the repo, returns source code, and mentions a 3-pass methodology. However, it is slightly ambiguous whether the methodology is executed by the tool or is a guide for the agent, and it does not mention side effects like repository size limits or auth needs. Still, the core behavior is transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, each serving a purpose: the first defines the core action, the second describes the workflow, and the third provides usage guidance. It is concise, front-loaded, and free of fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's purpose, when to use it, and the follow-up action, which is sufficient for a one-parameter tool with no output schema. However, it leaves some details ambiguous, such as the exact output format and how the 3-pass methodology manifests, but overall the context is well covered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema fully documents the single parameter 'source_url' with a description and example. The tool description does not add specific parameter-level details beyond what the schema already provides, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs a 'deep security audit of a Git repository' with a specific verb and resource. It distinguishes itself from siblings by explicitly mentioning check_package and submit_report, positioning this tool as the core auditing step.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance: 'Use check_package FIRST to see if an audit already exists — only use this for unaudited packages or when a fresh audit is requested.' It also instructs to call submit_report with findings after analysis, clearly defining when and how to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_packageA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description takes on the transparency burden. It discloses the tool's output (risk score, findings, status) and its not-found behavior (suggesting an audit), adding valuable behavioral context. However, it does not explicitly confirm that the operation is read-only or mention any side effects, though 'look up' strongly implies a non-destructive query.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured with the key purpose and usage priority up front. It delivers useful information in every sentence, though there is slight redundancy between 'USE THIS FIRST' and 'go-to tool' that could be tightened.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple lookup tool with one parameter and no output schema, the description covers all essential aspects: purpose, return values, usage timing, and fallback behavior. It is contextually complete for an agent to decide when and how to invoke it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema fully documents the single parameter package_name with a helpful description and examples (100% coverage). The tool description adds context about the package being an MCP server or package but does not provide additional parameter-specific semantics beyond the schema. A baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the action ('look up a package'), the resource ('AgentAudit security registry'), and the expected return info (risk score, findings, audit status). It also explicitly distinguishes itself from audit_package by positioning itself as the first-stop lookup tool, making it a clear purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use this tool ('USE THIS FIRST whenever the user wants to install, add, evaluate, or learn about a specific MCP server or package') and frames it as the go-to for safety questions. It also implies an alternative by noting that if the package isn't found, it suggests running an audit, which indirectly points to audit_package.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• submit_reportA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility for disclosing behavioral traits. It clearly states that 'The report becomes publicly available,' which is a critical irreversible side effect. However, it does not mention whether the submission is permanent or if any authentication is required, though the public-availability disclosure is the most important trait.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences, each earning its place: purpose, when to use, and why it matters. There is no redundancy or fluff, and the most important information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple interface (one parameter, fully documented) and no output schema, the description covers the essential context: the action, prerequisite, and public impact. It doesn't mention error handling or return values, but these are not required when the schema is self-sufficient. Minor gaps like potential validation or failure modes prevent a perfect score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides 100% coverage of the 'report' parameter, including required fields and their types. The description adds the concept of 'completed' and ties the report to the output of audit_package, but does not add further syntactic or structural detail. This aligns with the baseline of 3 for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Submit' with a clear resource: 'completed security audit report to the AgentAudit registry (agentaudit.dev)'. This distinguishes it from sibling tools like audit_package and check_package, which are about analysis rather than submission. The mention of being 'after audit_package' further clarifies its role.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives an explicit when-to-use instruction: 'Call this after you have analyzed the code from audit_package.' This is a clear prerequisite that helps the agent choose the correct timing. It also implies not to call before an audit is complete, effectively excluding premature use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• discover_serversA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It clearly implies a read-only operation by using 'scan' and 'list', and specifies the source as local config files. It does not explicitly state 'read-only' or mention any side effects, but the verbs make the passive nature clear. Lacks detail on what happens after scanning (e.g., output format), but the core behavior is transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tightly packed sentences: the first states the tool's purpose and scope, the second provides usage boundaries. No fluff or redundancy. Front-loaded with the primary action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (one optional parameter, no output schema, no annotations). The description fully covers what the tool does, when to use it, and when not to. It even names the alternative. The lack of an output schema is mitigated by the description's 'list' wording, which implies the return type.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (the single parameter check_registry is described in the schema). The description adds no additional semantics about the parameter, but the baseline is 3 when schema coverage is high, and the parameter's purpose is clear from the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('list') and resource ('ALREADY INSTALLED MCP servers') with explicit sources (Claude Desktop, Cursor, Windsurf, VS Code). It clearly distinguishes from sibling tools by instructing to use check_package for install/evaluate/lookup tasks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use ('Use ONLY when the user wants to review/list their existing servers') and when-not-to-use ('Do NOT use this when the user wants to install, evaluate, or look up a specific package'), and names the alternative tool (check_package). This is exemplary guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.