Server Quality Checklist

Tool Scores

• scan_repositoryB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It mentions safety features (path resolution via security/paths.ts, bounded by ResourceGovernor) and return type (typed findings). However, it does not disclose potential side effects, error behavior, or destructive nature beyond the safe-by-default claim.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (two sentences) and front-loaded with the core action. However, the lack of parameter details makes it feel incomplete, slightly reducing efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool complexity (6 parameters, no output schema, no annotations), the description is insufficient. It covers purpose and safety but omits parameter semantics, return value structure, and usage guidelines, leaving the agent underinformed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. Despite six parameters including 'repoRoot', 'includeHidden', 'maxFiles', etc., the description provides no explanation of their meaning, defaults, or valid values. This leaves the AI agent without the information needed to set parameters correctly.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (walk, parse, return) and resource (repository root) and specifies three types of findings (vulnerabilities, secrets, malformed inputs). It distinguishes the tool from siblings like audit_dependencies and generate_sbom.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for general repository scanning but does not explicitly state when to use it vs alternatives or provide exclusion criteria. Lacks guidance on prerequisites or context-specific usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• audit_dependenciesA
  3.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Describes mock vs live adapter behavior (network, retry, cache, rate limits) and return value, compensating for missing annotations. Does not cover authentication or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, no fluff, front-loaded with main action and key details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers adapters and return but lacks explanation of repoRoot path requirements, ecosystem options, and output format. Adequate for simplicity but gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Adds meaning to osvMode by explaining mock/live behavior; ecosystem and repoRoot are not described. With 0% schema coverage, description partially compensates.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool audits repository dependencies against OSV and lists adapters and outputs. It distinguishes from siblings by context, though not explicitly.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives like scan_repository or generate_sbom. Missing prerequisites or scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_sbomA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses output format (CycloneDX 1.5), included fields, and validation step. Lacks details on potential network access, performance impact, or any side effects, but adequate for a read-only generation tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with main action, no redundant words. Efficient and clear.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Missing explicit return value or output format (e.g., JSON/XML). No output schema to compensate. Does not explain repoRoot parameter's role or constraints.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No description adds meaning to the two parameters. Schema provides type/enum only; repoRoot has no explanation of its purpose or semantics. With 0% coverage, description fails to compensate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clear verb 'Build' and resource 'CycloneDX 1.5 SBOM for the repository' with specific format and component details. Distinct from siblings like audit_dependencies and scan_repository.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies use when needing an SBOM, but no explicit guidance on when to use this vs. alternatives like audit_dependencies or scan_repository. No exclusions or prerequisites mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• generate_evidence_reportA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description effectively discloses output behavior: JSON content fields and HTML self-contained features. It does not mention destructive actions or side effects, but for a report generation tool this is reasonable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (two sentences) and front-loaded with the core purpose. However, the second sentence is dense with details that could be more structured. Still, every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and zero parameter descriptions, the description leaves significant gaps. It provides output details but not input semantics or process context, making it only partially complete for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description does not explain any parameters (repoRoot, format, includeHtmlPreview). Agents must infer or rely on parameter names alone, which is insufficient for correct usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose ('Assemble the final evidence report') and distinguishes it from sibling tools (audit_dependencies, generate_sbom, scan_repository) by specifying that it produces the final report with detailed content descriptions for JSON and HTML forms.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is the final step ('final evidence report') but provides no explicit guidance on when to use it versus alternatives or prerequisites. It does not mention that this tool should be used after other scanning/auditing tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.