Server Quality Checklist

Tool Scores

• search_vendorsA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the output (vulnerability counts) but does not explain query behavior (e.g., matching semantics, case sensitivity), pagination via limit/offset, or what happens with no results. This is minimal behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that conveys the core purpose and output without unnecessary words. It is concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema and only 3 parameters, the description is minimally complete for a search tool. However, it lacks context on when to use it versus get_vendor_overview, and it does not specify pagination behavior or query nuance. The presence of an output schema helps, but the description still leaves gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains the 'query' parameter can contain name, category, or headquarters, adding meaning. However, it does not clarify the relationship between these search criteria (e.g., separate fields vs. combined search) or explain limit/offset, though those have schema defaults and constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Find' and clearly identifies the resource (vendors). It specifies search criteria (name, category, or headquarters) and the return value (vulnerability counts). This differentiates it from siblings like search_vulnerabilities and get_vendor_overview.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (when searching for vendors by attributes), but it does not provide explicit alternatives or exclusions. It does not contrast with get_vendor_overview or search_vulnerabilities, leaving some ambiguity for the agent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_vulnerability_statisticsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses a key behavioral trait: 'without returning every matching record', implying aggregate output and non-destructive reads. However, it does not explain aggregation specifics (e.g., grouping, pagination) or any permissions/limits. This is adequate but not thorough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, with the primary action and resource stated immediately. It avoids fluff and every sentence contributes: the first defines behavior, the second gives a use-case example.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present, return values need not be described. The description gives a clear purpose and one usage example, but with no annotations and six parameters (all optional) lacking schema descriptions, the tool definition feels slightly under-specified for a fully self-contained agent evaluation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, but it only implicitly references 'critical' and 'open' parameters. Parameter names and enums in the schema provide some self-evident meaning, but the description adds virtually no parameter-level detail beyond what the schema already shows.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses specific verbs 'Count and summarize' with the resource 'vulnerabilities', clearly distinguishing this aggregation tool from sibling tools like get_vulnerability and search_vulnerabilities, which return records. It also gives an example question ('how many critical vulnerabilities are open') that reinforces the purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly states when to use this tool: for questions requiring counts or summaries rather than full matching records. It provides a concrete example, but does not explicitly name alternatives or state when not to use it, so it misses the bar for a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_vulnerabilitiesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It implies a read-only operation through 'Search and filter,' but does not explicitly state that it does not modify data, nor does it mention pagination behavior or return format (though output schema exists). The description adds some context beyond the schema (e.g., returns lists) but lacks depth about side effects or edge cases.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exactly two sentences: the first states the action and filter dimensions, the second clarifies the use case. Every word earns its place with no redundancy or irrelevant detail, making it highly efficient and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (13 optional parameters, many enums, output schema), the description covers the essential context: the purpose, main filter categories, and the fact that it returns lists. The output schema covers return-value structure, and the filter dimensions are enumerated. It lacks explicit details about default pagination or search semantics, but this is adequately handled by the schema and the concise use-case note.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It groups parameters into conceptual filters (text, vendor, severity, status, CVSS score, publication date), which adds high-level meaning. However, it stops short of mapping each filter to specific parameter names (e.g., 'text' → query) or explaining nuances like date range pairs, so it only partially clarifies the 13 parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Search and filter') and identifies the resource ('vulnerabilities') with an explicit list of filter dimensions (text, vendor, severity, status, CVSS score, publication date). The phrase 'Use this for lists of matching CVEs' distinguishes it from sibling tools like get_vulnerability, indicating this is for list retrieval rather than single-record lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context by stating 'Use this for lists of matching CVEs,' which tells the agent when to select this tool. It does not explicitly name alternatives or exclude cases, but the list-focused guidance is sufficient given the sibling tool names (e.g., get_vulnerability). No misleading or contradictory usage advice is present.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_vulnerabilityA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behavior. It states it retrieves data but does not mention error handling, authentication, rate limits, or what happens if the CVE is not found. The list of fields provides some context about the response, but the behavioral surface is minimal. Given the simplicity, a score of 3 is reasonable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary action and constraint. Every word adds value, with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the single required parameter and an existing output schema, the description sufficiently explains when to use the tool and what data it returns. The mention of specific fields and exact CVE identifier covers the prerequisites. The output schema handles return structure.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides a full description of cveId with an example. The tool description only restates 'exact CVE identifier,' adding no new semantic information. Since schema coverage is 100%, the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action ('Retrieve one vulnerability') and the key constraint ('by its exact CVE identifier'), distinguishing it from sibling search_vulnerabilities. It also lists the specific fields the caller can access, which reinforces the purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for a single known CVE, saying 'exact CVE identifier' and listing details like score, status, and vendor. However, it does not explicitly mention alternative tools or when not to use it, though the word 'exact' suggests searching is for unknown CVEs. This is a clear context but lacks explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_vendor_overviewA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It transparently describes what the tool returns: one vendor, aggregate vulnerability statistics, and the five most recent vulnerabilities. It doesn't mention error behavior or permissions, but for a simple read operation, the disclosed behavior is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is just two sentences, front-loading the primary action in the first sentence and usage context in the second. Every word contributes value; there is no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity and the presence of an output schema, the description covers essential aspects: what is returned (vendor, stats, recent vulns) and when to use it (after vendor ID identification). It doesn't cover edge cases like not-found behavior, but the usage instruction and output schema make the description sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only a 'vendorId' string with no description (0% coverage), so the description must compensate. The phrase 'Use after identifying a vendor ID' clarifies that vendorId is the identifier of the vendor to fetch, which is sufficient for a single-parameter tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description begins with 'Return one vendor, aggregate vulnerability statistics, and its five most recent vulnerabilities,' which clearly states the tool's function with a specific verb and resource. It also distinguishes itself from sibling tools like search_vendors and get_vulnerability by combining vendor details with statistics and recent vulnerabilities.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Use after identifying a vendor ID' provides an explicit precondition for when to invoke this tool. While it doesn't explicitly contrast with siblings, the context signals and the instruction make it clear that this is for post-identification overview rather than discovery or individual vulnerability lookup.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.