Server Quality Checklist

Tool Scores

• pci_list_scansA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, so the safety profile is known. The description adds behavioral context about ordering ('newest first') and pagination via offset/limit, but does not disclose other traits like default page size, maximum page size beyond the limit range, or what fields are returned. This is acceptable for a straightforward read-only list tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and well-structured. The purpose is front-loaded in a single sentence, and the parameter documentation is presented as a clean list. Every sentence earns its place, with no filler or redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list operation with two optional parameters, the description is reasonably complete. Annotations cover read-only safety, an output schema exists, and the parameter semantics are defined. It could benefit from a brief note on when to use pci_list_scans versus pci_get_scan, but the overall information is sufficient for basic use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate. It does so by documenting both parameters: 'offset: 1-based record offset' and 'limit: Rows per page, 1-1000.' This provides clear meaning beyond the bare schema, though it omits details about default behavior when parameters are null.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'List PCI scans, newest first.' The verb 'list' and resource 'PCI scans' are specific, and the ordering 'newest first' adds useful detail. This distinguishes it from siblings like pci_get_scan (which retrieves a single scan) and pci_list_assets (which lists a different resource).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for enumerating scans but does not explicitly state when to use it over alternatives like pci_get_scan or pci_list_vulns. There are no exclusions or comparisons to sibling tools. The context of sibling names provides some guidance, but the description itself lacks direct usage recommendations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pci_get_vulnA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare the tool as read-only and non-destructive, so the description doesn't need to repeat that. It adds useful behavioral context by stating the response includes 'full detail' and 'prose fields,' and clarifies the relationship to pci_list_vulns. No contradictions with annotations are present.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and front-loaded: the first sentence states the purpose, and the second provides the essential parameter guidance. Every word earns its place, with no fluff or repetition of schema details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a single required parameter, an output schema, and read-only annotations, the description covers the essential usage context. It could optionally mention when to use this over pci_get_scan, but for a focused detail-retrieval tool with good schema support, it is nearly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With schema description coverage at 0%, the description fully compensates by explaining the only parameter, detection_id, as 'The finding's `id` from pci_list_vulns -- not its `qid`.' This adds critical semantic meaning beyond the raw integer schema and prevents a common misuse.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool fetches 'Full detail for one vulnerability finding, including prose fields,' which specifies both the verb (get) and the resource (a single vulnerability finding). It distinguishes itself from siblings like pci_list_vulns by emphasizing 'one vulnerability finding' and cross-referencing the list tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly instructs that detection_id should be taken from pci_list_vulns and not the qid, providing clear guidance on the correct ID source. While it doesn't enumerate alternatives or exclusions beyond the qid distinction, this is sufficient for a simple get-by-id tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pci_compliance_summaryA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, so the safety profile is known. The description adds valuable behavioral details beyond annotations: pagination semantics (offset 1-based, limit 1-1000, fetch_all), the deliberate omission of an ip filter due to a Qualys bug, and the presence of total_count for scope visibility. These are not redundant with annotations, giving solid transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: a clear one-line purpose, an Arguments list, and a brief caveat about the ip filter. Every sentence earns its place—no fluff. The layout is scannable and front-loaded with the core intent. It is concise but complete for its content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 5 parameters, 0% schema description coverage, and no parameter-level documentation, the description covers all necessary context: pagination, filtering, the missing ip filter, and total_count. An output schema exists, so extra return-value explanation is not needed. The description is fully adequate for an agent to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description is the sole source of parameter meaning. It explains every parameter (offset, limit, dns, compliance_status, fetch_all) including ranges and purpose. It also explains why a parameter (ip) is missing, which is crucial for understanding behavior. This fully compensates for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Per-host PCI Pass/Fail status' which clearly identifies the resource and the nature of the data. The phrase 'The primary way to iterate hostnames' further specifies its role and distinguishes it from sibling tools that list assets, vulns, or scans.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly calls this 'The primary way to iterate hostnames,' giving clear when-to-use context. It also explains why there is no ip filter, providing a useful usage caveat. It does not explicitly mention alternative sibling tools, but the 'primary' designation and lack of ambiguity justify a 4.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pci_get_scanA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and destructiveHint. The description adds a limitation: active_hosts is aggregate and no per-host breakdown is reported. This is valuable behavioral context not captured by annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (two short sentences plus argument spec). It front-loads the purpose and every sentence adds value, including the limitation note.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema available, the description adequately covers the single parameter and a key data limitation. The tool is simple (one param, no nested objects) and the description is complete for an agent to select and invoke it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 0% description coverage, but the description fully compensates by explaining scan_id as The scan's scan_id from pci_list_scans, providing provenance and meaning beyond the integer type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Detail for one scan', clearly specifying a verb (get) and resource (scan). It distinguishes from siblings like pci_list_scans by focusing on a single scan's detail and explicitly references the aggregate active_hosts count.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by instructing scan_id from pci_list_scans, indicating this is a follow-up tool. It doesn't explicitly name alternatives, but the sibling names and the 'one scan' phrasing provide clear context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pci_list_assetsA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false. The description adds valuable context about the data being configuration rather than observation, which clarifies the meaning of the output. This goes beyond the annotations without contradicting them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with the purpose. The arg list and clarifying note are directly relevant, and every sentence earns its place without unnecessary filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple one-parameter tool with an output schema and readOnly annotation, the description covers the purpose, parameter values, and the key nuance about declared vs observed scope. It is complete for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides no description for asset_type (0% schema description coverage), but the description explicitly lists the allowed values: ALL, IP, DNS, or VIRTUALHOST. This fully compensates for the schema gap and adds meaning to the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'List declared in-scope PCI assets', which is a specific verb+resource. It also distinguishes itself from sibling tools like pci_list_scans and pci_list_vulns by focusing on declared assets rather than scan results.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: 'declared scope -- configuration, not observation' and explicitly notes it 'says nothing about whether a host responded to a scan.' This helps the agent know when to use this tool (to get declared assets) and not to use it for scan observations, but it does not name alternative tools explicitly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pci_list_vulnsA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and destructiveHint=false, so the safety profile is known. The description adds behavioral context beyond that: it returns 'compact rows' with no prose fields, explains fetch_all pagination behavior ('Walk every page rather than returning one'), and lists filter semantics. This enriches the agent's understanding of what to expect.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is tight and scannable: a one-line summary followed by a bullet-like list of parameters, each with a succinct definition. The final sentence about prose fields and pci_get_vuln is valuable and not redundant. No noise or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having 10 parameters and no required ones, the description covers all of them with functional meanings. It also explains the output style (compact rows, no prose) and points to the sibling tool for details. Since an output schema exists, return-value details are not needed. The tool description is complete for effective usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description carries full responsibility for parameter meaning. The Args section provides a concise, meaningful explanation for each of the 10 parameters, including allowed values for severity and false_positive. This goes beyond the schema, which only provides types and defaults, and gives the agent enough to construct correct calls.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear verb and resource: 'List vulnerability findings as compact rows.' It distinguishes itself from sibling tools like pci_get_vuln (which provides detailed prose for a single finding) and pci_list_assets (which lists assets). The function's scope is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use: for listing/filtering vulnerability findings in a compact format. It explicitly states an alternative: 'call pci_get_vuln for one finding's threat, impact, and solution text,' indicating when to switch. However, it does not contrast with other list tools like pci_list_assets or pci_list_scans, leaving some usage context implicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• pci_download_scan_reportA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Even with annotations declaring readOnlyHint=true and destructiveHint=false, the description adds significant behavioral detail: bytes are written to disk, only a path is returned, a 404 is normal and not a failure, and it explicitly warns not to treat a 404 as a compliance finding. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, then gives parameter details, then essential caveats. Every sentence contributes value, and the bold formatting for the 404 warning makes it stand out. No redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter, the description covers prerequisites (generation in console), output behavior (path to saved PDF), error semantics (404), and operational guidance (do not report as compliance finding). It is highly complete given its scope.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has one parameter (scan_id) with no description (0% coverage). The description compensates by explaining that scan_id comes from pci_list_scans, providing provenance that is absent from the schema. This adds meaning beyond the structured field.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Download') and resource ('a scan's report PDF') and the outcome ('return the path'). It is immediately distinguishable from sibling tools that list or get scans/assets/vulns.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context: PDF is the only format, reports must be generated in the console first, and a 404 is a normal condition indicating no report has been generated. It does not explicitly name alternatives, but the context strongly implies when to use and what to expect.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.