Server Quality Checklist

Tool Scores

• codeinspectus_generate_sbomA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond annotations, the description adds that it writes a file, returns location and component count, and operates offline. This provides valuable behavioral context not present in the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, front-loaded with purpose, and uses only three short sentences without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity and the presence of an output schema, the description covers the key aspects: what it does, output details, and offline nature. It lacks only minor context like prerequisites.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so parameters are well documented. The description mentions the format and output path behavior, but adds minimal extra meaning beyond the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifies generating CycloneDX or SPDX SBOM using Trivy, clearly distinguishing it from sibling tools like codeinspectus_scan or codeinspectus_compliance_report.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states 'Offline' and describes output behavior, but does not explicitly compare to siblings or state when to use this tool versus alternatives. Usage is implied but not fully clarified.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codeinspectus_scanA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint. The description adds important behavioral details: 'Fully offline — zero network egress at scan time' and 'Never writes to your code or repo', which reinforce safety and align with annotations. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (two sentences) but packs in key information: bundled engines, AI checks, output format, offline mode, and no-write guarantee. It is front-loaded with the main purpose. Slightly verbose for the sibling context, but well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 params, output schema, annotations), the description covers all essential aspects: what it does, what engines it uses, output characteristics, offline safety, and non-destructiveness. The output schema handles return format details. No gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All 5 parameters are fully described in the schema (100% coverage). The description adds minimal parameter-specific meaning beyond the schema, mentioning 'path' and 'scanners' implicitly but not adding new details. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it runs a full local security scan using multiple engines (Opengrep SAST, Gitleaks, Trivy, AI checks) and returns CWE-keyed findings. It distinguishes itself from siblings like compliance_report or explain_finding by focusing on scanning and including a list of specific scanners.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions 'full local security scan' and 'Fully offline' implying it's for comprehensive offline scanning, but it does not explicitly say when to use this tool over alternatives like compliance_report or rescan. No usage exclusions are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codeinspectus_rescanA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false. The description adds value by stating 'Never writes to your code or repo' and explaining the output includes resolved, remaining, and new findings, enhancing transparency beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three concise sentences, front-loaded with the core purpose, and contains no unnecessary words. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains the core function and output categories. With an output schema present, return values are covered. However, it does not mention optional filtering parameters (scanners, severity_threshold), which could help agents understand customization options.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 40%, leaving 3 of 5 parameters (scanners, max_findings, severity_threshold) undocumented in the schema. The description does not provide additional details for these parameters, failing to compensate for the gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool re-runs a scan and diffs against a prior scan, reporting resolved, remaining, and new findings. It explicitly distinguishes from the sibling 'codeinspectus_scan' by focusing on re-scanning after fixes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states 'Use this to verify fixes' and 'Never writes to your code or repo', providing clear context for safe use. It implicitly contrasts with initial scanning but does not explicitly exhaust alternatives or when not to use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codeinspectus_compliance_reportA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint, idempotentHint, and non-destructive. The description adds valuable context: it is 'code-level evidence only' and not an audit, and describes the output format ('X of N code-visible controls have findings' with explicit denominator). This goes beyond annotations to set correct expectations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core action and resource, and includes a clear caveat. No extraneous information; every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema, complete input schema, and annotations, the description sufficiently covers the tool's purpose, input requirements, output format hint, and limitations. It does not need to explain return values since output schema exists, and for a read-only reporting tool, this is complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema covers 100% of parameters with descriptions, so the burden on the description is low. The description mentions the frameworks in parentheses, mirroring the enum, and explains the per-framework view, which aligns with the optional framework parameter. This adds minimal extra meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool produces a 'per-framework code-level control-coverage view for a prior scan,' specifies the supported frameworks, and explicitly distinguishes itself from a compliance audit. The verb 'produce' and resource 'code-level control-coverage view' are specific. Sibling tools like codeinspectus_explain_finding have different purposes, so this tool is well-differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description notes this tool is for a 'prior scan,' implying it should be used after a scan. It also warns that it is NOT a compliance audit or attestation, guiding against misuse. However, it does not explicitly state when to use this tool over specific siblings like codeinspectus_explain_finding, leaving some ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codeinspectus_explain_findingA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnlyHint, idempotentHint, destructiveHint false, indicating safe, non-destructive behavior. The description adds value by detailing what the explanation covers (weakness, remediation steps, references), which aligns with annotations. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that immediately states the tool's purpose and output, with no superfluous information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple lookup tool with an output schema present, the description sufficiently covers purpose, inputs, and output quality. No additional details about return values are needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema has 100% coverage; both parameters (scan_id, finding_id) are described with constraints and descriptions. The description does not add further semantic detail beyond the schema, meeting the baseline expectation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns a deep explanation and remediation plan for a single finding from a prior scan, specifying the content (weakness, importance, fix steps, references). It is distinct from siblings like codeinspectus_scan or codeinspectus_list_rules.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description indicates use when a specific finding id from a prior scan needs explanation. It implicitly suggests prerequisites (having scan results) but does not explicitly list cases to avoid or alternatives. Siblings are differentiated by tool name and purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codeinspectus_list_rulesA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint. The description adds value beyond annotations by specifying exactly which types of information are listed (e.g., Trivy vulnerability-DB freshness, custom AI code rules). No contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, tightly packed sentence with no wasted words. Every element (detectors, engine versions, DB versions, AI code rules) is listed efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given low complexity (1 optional parameter, no required params) and the presence of an output schema, the description sufficiently conveys the tool's scope. No need to detail return values as output schema exists.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with a clear enum and description for the single parameter. The description does not add additional meaning beyond the schema, so baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses specific verbs and resources, listing exactly what is returned (active detectors, engine versions, database versions, etc.). It clearly distinguishes this tool from sibling tools like codeinspectus_scan or codeinspectus_compliance_report.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for inspecting current rules and versions but does not explicitly state when to use this tool versus alternatives or provide exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.