Server Quality Checklist

Tool Scores

• analyze_licensesB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It states analysis is for 'completed scans', implying it requires a finished scan, but does not mention if the tool is read-only, if it returns output, or if any side effects occur. With no annotations, this is insufficient transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that is concise and front-loaded with the main purpose. It earns its place, though it could be slightly more expansive without becoming verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has only one parameter and no output schema, the description is minimally adequate. It states the function and scope, but lacks details on return values, required scan state, and differentiation from related tools. It is complete enough for a simple tool but has gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the single parameter 'job_id' is described minimally in the schema as 'The scan job ID'. The description adds no further meaning beyond what the schema provides, but with 100% coverage, baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool analyzes license compliance for completed scans, and specifies it identifies copyleft, restrictive, and unknown licenses. It uses a specific verb and resource, and distinguishes from siblings like check_package or get_findings, though not explicitly contrasting.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool should be used after a scan is completed, but does not explicitly tell when to use it vs alternatives, nor does it mention prerequisites like the job_id must refer to a completed scan. No exclusions or when-not-to-use guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_scansB
  3.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description implies a read-only operation (comparing existing data), which aligns with the absence of destructive annotations. However, it does not disclose any limits on scan size, performance considerations, or whether the results include all findings or a summary.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that efficiently covers the core purpose and the key outcome categories. It is front-loaded and contains no filler, but could be slightly more structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 2 parameters with 100% schema coverage, no output schema, and no annotations, the description provides a basic understanding but lacks details on return format, error handling, or limitations. It is minimally adequate but not fully comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Both parameters are described in the input schema with clear descriptions ('First scan job ID (baseline)' and 'Second scan job ID (current)'), and coverage is 100%. The description adds nothing beyond the schema, so a baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool compares vulnerability findings between two scans and explains the three possible outcomes (added, removed, unchanged). It uses a specific verb ('compare') and resource ('vulnerability findings'), though it could better differentiate from sibling tools like 'get_findings'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives such as 'get_findings' or 'list_scans'. There is no mention of prerequisites (e.g., both scans must exist) or situations when another tool would be more appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_cveB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears the burden. It indicates a lookup operation (likely read-only) but does not disclose any behavioral traits like data source, response format, or potential limitations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that clearly communicates the tool's purpose with no unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low complexity (single parameter, no output schema), the description is minimally adequate. However, it lacks details on what 'details' are returned or any error conditions.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single parameter. The description adds a usage example but does not provide additional meaning beyond the schema's description of the CVE identifier format.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that the tool looks up details about a specific CVE vulnerability. The verb 'look up' and resource 'CVE vulnerability' are specific. Among siblings, it is distinct as it deals with CVE lookup, not package scanning or license analysis.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not provide guidance on when to use this tool vs alternatives. No context about prerequisites or when not to use it is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_findingsB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description must carry burden. It describes the output content but does not mention pagination limits, rate limits, or whether it can be called on in-progress scans. Assumed read-only, but no explicit statement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with purpose. First sentence states action and resource, second lists return fields. Efficient with no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a retrieval tool with no output schema, description covers basic return fields and required parameter. Lacks information on error conditions (e.g., invalid job_id) or performance characteristics. Adequate but not comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. Description adds meaning for job_id (required) and severity (filter), but does not explain pagination semantics beyond schema defaults.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns detailed vulnerability findings for completed scans, specifying CVE IDs, severity, affected packages, and descriptions. This differentiates it from siblings like search_cve (search by CVE) and list_scans (list scans).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use after a scan is completed, but does not explicitly state when to use this vs. filtering via search_cve or compare_scans. No exclusions or alternatives are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_statusA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description indicates this is a read-only status check, which is consistent with typical behavior. However, since annotations are missing, the description carries the full burden; it could benefit from stating whether the tool is non-destructive, or if it has any side effects, but it is not misleading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, concise sentence with no unnecessary words. It front-loads the purpose and is immediate in delivering the core functionality.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool is simple (one required parameter) and the schema covers the parameter well, the description is mostly adequate. It could mention that the status is returned or what format the results are in, but the low complexity reduces the need for more detail.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% for the single parameter 'job_id', which is clearly described as 'The scan job ID returned by scan_image'. The description does not add extra meaning beyond the schema, so baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('check') and resource ('status and results of a vulnerability scan'), which aligns with the tool's name. It distinguishes from siblings like 'scan_image' (which initiates a scan) and 'get_findings' (which retrieves detailed findings), though it does not explicitly call out the distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool should be used after initiating a scan with 'scan_image' (as it requires job_id from that tool). However, it does not provide explicit when-to-use or when-not-to-use guidance, nor does it mention alternatives among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_scansA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description mentions it lists 'recent' scans, indicating some ordering, but does not disclose other behavioral traits like whether it returns scans across all users or only those initiated by the current user, or any rate limiting. Since no annotations are provided, the description partially fulfills transparency but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear, front-loaded sentence with zero unnecessary words. It effectively conveys the tool's purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool is simple with one optional parameter and no output schema, the description provides adequate completeness. However, it could mention the return format (e.g., list of scan objects) to be fully self-contained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already fully describes the 'limit' parameter with a description and default value. The description does not add additional meaning beyond the schema, but given 100% schema coverage, a baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'list', the resource 'recent vulnerability scans', and the scope 'for your organization'. It is specific and distinguishes itself from siblings like 'get_findings' and 'scan_status'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when a user wants to see recent scans for their organization, but it does not provide explicit guidance on when to use this versus alternatives such as 'scan_status' or 'compare_scans'. No exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_packageA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It states it uses the OSV database but does not disclose behavioral traits like whether it makes external API calls, rate limits, or whether it's a read-only operation. Annotations would help, but the description is adequate for understanding the core behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence that conveys the tool's purpose without any filler. Highly concise and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the moderate complexity (3 params, no output schema, no annotations), the description is complete enough for basic usage. It doesn't explain return value structure, but in the absence of an output schema, a bit more detail (e.g., 'returns list of vulnerabilities') would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents all three parameters. The description adds no additional information beyond what's in the schema (e.g., that version is optional and omitting checks all versions). Thus it meets the baseline but does not exceed it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool checks a specific package version for known vulnerabilities using the OSV database. It uses a specific verb ('Check') and resource ('package version'), and distinguishes well from siblings like analyze_licenses or search_cve, which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you need to check vulnerabilities for a specific package version. It does not explicitly state when not to use it or mention alternatives, but the context is clear given the sibling tools (e.g., search_cve for broader CVE searches).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• scan_imageA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It effectively discloses that scanning is asynchronous by returning a job ID, and implies that results need to be fetched later via another tool (scan_status). This is good behavioral transparency for a non-annotated tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, concise and front-loaded: first sentence defines purpose and resource, second sentence adds key behavioral note about job ID. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (2 params, no output schema), the description is nearly complete. It covers purpose, async nature, and job ID tracking. However, it could mention that the job ID is used with scan_status or get_findings, but that is implicit from sibling names.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description covers both parameters (100% coverage). The description adds no extra meaning beyond the schema, so baseline is 3. However, the description uses the tool's purpose to imply that the image parameter is the primary input, and mentions mode options, earning a 4 due to clarity in context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool scans Docker/OCI images for vulnerabilities, with a specific verb 'scan' and resource 'Docker/OCI container image'. It also notes it returns a job ID for tracking, distinguishing it from sibling tools like scan_status which presumably retrieves results.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for vulnerability scanning but does not explicitly state when to use this tool vs alternatives like check_package or scan_status. Sibling tools exist but no guidance on when to choose scan_image over them is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.