Server Quality Checklist

Tool Scores

• query_for_cve_affectedB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the basic operation but lacks critical details: it doesn't mention error handling (e.g., what happens if the CVE isn't found), rate limits, authentication needs, or data freshness. For a query tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and concise: a clear purpose statement followed by 'Args' and 'Returns' sections. Every sentence adds value without redundancy, and it's front-loaded with the core functionality. The brevity is appropriate for a simple query tool.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (one parameter, no output schema, no annotations), the description is minimally complete. It covers the basic purpose and parameter semantics but lacks behavioral details and usage guidelines. Without an output schema, it hints at the return type ('A list of affected versions') but doesn't specify the structure (e.g., version ranges, packages). This is adequate but has clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful context for the single parameter: it explains that 'cve' is 'The CVE ID to query,' which clarifies the expected format beyond the schema's basic 'string' type. With 0% schema description coverage and only one parameter, this adequately compensates, though it could specify format examples (e.g., 'CVE-2021-12345').

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Query the OSV database for a CVE and return the affected versions.' This specifies the verb ('query'), resource ('OSV database'), and outcome ('return the affected versions'). However, it doesn't explicitly differentiate from sibling tools like 'query_for_cve_fix_versions' or 'query_package_cve', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With sibling tools like 'query_for_cve_fix_versions' (which likely returns fix versions) and 'query_package_cve' (which might query by package), there's no indication of when this specific CVE-affected-versions query is appropriate. The minimal context ('Query the OSV database for a CVE') is insufficient for distinguishing use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_for_cve_fix_versionsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. While it describes the basic operation (querying and returning fix versions), it doesn't mention important behavioral aspects like error handling, rate limits, authentication requirements, or what happens with invalid CVE inputs. For a tool with zero annotation coverage, this leaves significant gaps in understanding its behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and well-structured. It uses exactly three sentences: one for the purpose, one for the parameter, and one for the return value. Every sentence earns its place with no wasted words. The information is front-loaded with the core purpose stated first.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (querying an external database), no annotations, no output schema, and 0% schema description coverage, the description is minimally adequate. It covers the basic purpose and parameter but lacks important context about error conditions, return format details, or behavioral constraints. The absence of output schema means the description should ideally explain more about the return value structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description explicitly documents the single parameter ('cve: The CVE ID to query'), adding meaningful context beyond the schema. With 0% schema description coverage and only one parameter, the description fully compensates by explaining what the parameter represents. The baseline would be 3 with good schema coverage, but here the description provides complete parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Query the OSV database for a CVE and return the fix versions.' It specifies the verb ('query'), resource ('OSV database'), and outcome ('return the fix versions'). However, it doesn't explicitly differentiate from sibling tools like 'query_for_cve_affected' or 'query_package_cve', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With sibling tools like 'query_for_cve_affected' and 'query_package_cve' available, there's no indication of when this specific query for fix versions is appropriate versus those other queries. The description lacks any 'when-to-use' or 'when-not-to-use' context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• query_package_cveA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions the tool queries a database and returns CVE IDs, but lacks details on rate limits, error handling, authentication needs, or what happens if no CVE is found. For a security-related query tool with zero annotation coverage, this is a significant gap in behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear purpose statement, usage guidelines, and parameter explanations. It's front-loaded with key information and uses bullet points for parameters. Some redundancy exists (e.g., repeating 'query' in the first two sentences), but overall it's efficient and easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, no output schema, and 3 parameters with 0% schema coverage, the description does a decent job by explaining parameters and usage. However, it lacks details on return format (e.g., structure of the list of CVE IDs), error cases, or performance considerations, leaving gaps for a tool with moderate complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description compensates well by explaining each parameter's purpose: 'package' is the name, 'version' can be None for all versions, and 'ecosystem' can be None for all ecosystems, with a reference to 'get_ecosystems' for supported options. It adds meaningful context beyond the bare schema, though it doesn't cover all possible edge cases.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Query the OSV database for a package and return the CVE ID.' It specifies the verb ('query'), resource ('OSV database'), and output ('CVE ID'), making it easy to understand. However, it doesn't explicitly differentiate from sibling tools like 'query_for_cve_affected' or 'query_for_cve_fix_versions', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit usage guidance: 'ALWAYS use it before installing packages to check if the package is vulnerable' and gives examples (e.g., requirements.txt). It also mentions using it before updating packages and references the 'get_ecosystems' tool for ecosystem details, offering clear context and alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_ecosystemsA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It successfully describes the return format ('Dictionary of ecosystems with the key being the ecosystem name and the value being the programming language / OS') and a critical behavioral constraint ('you must use the exact ecosystem name as it is in the dictionary'). However, it doesn't mention potential limitations like rate limits or authentication requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly structured with three focused sentences: purpose definition, return format specification, and usage guidance. Every sentence adds essential information with zero redundancy or unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter tool with no annotations and no output schema, the description provides excellent context about the return format and how to use the output with sibling tools. The only minor gap is the lack of information about potential error conditions or system limitations, but this is reasonable given the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters with 100% schema description coverage, so the baseline is 4. The description appropriately doesn't waste space discussing nonexistent parameters, maintaining focus on the tool's purpose and output.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verb ('Get') and resource ('all supported ecosystems'), and provides a precise definition of what an ecosystem is ('package manager or repository for a specific language'). It distinguishes this from sibling tools by focusing on ecosystem retrieval rather than CVE queries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use this tool ('To use the ecosystem name in the query_package_cve function') and provides a clear alternative context by naming the specific sibling tool where the output is needed. It establishes a prerequisite relationship between tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.