Server Quality Checklist

Tool Scores

• codeinspectus_scanA
  3.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only and non-destructive behavior, and the description adds concrete guarantees: 'Fully offline — zero network egress at scan time' and 'Never writes to your code or repo.' It also explicitly discloses unavailable capabilities such as AI attribution, C2PA, and statistical-watermark support, which is valuable beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the primary action and then organized into scope, return types, limitations, and safety guarantees. It is dense and long, but the complexity of the tool justifies most of the detail. The V3 trust-contract sentence is somewhat specialized and could be trimmed, but overall every major block earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex tool, the description covers the scan scope, output categories, offline behavior, write-safety, and unavailable capabilities. An output schema exists, so detailed return-value documentation is not necessary. The main missing context is an explicit prerequisite note about codeinspectus_setup or guidance on when to use rescan/compliance alternatives, which keeps this from a 5.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% parameter description coverage, so the baseline is 3. The description does not add meaningful per-parameter semantics beyond the schema; it mentions scanner categories and output types, but the schema already documents path, scanners, max_findings, include_compliance, and severity_threshold. No contradiction or major gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a clear verb and resource: 'Run a full local security scan of a path.' It is unambiguous about what the tool does and enumerates the scan categories. It doesn't explicitly name sibling tools for differentiation, but the word 'full' and the specific output details distinguish it enough from the related compliance/report/SBOM tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description says what the tool does but gives no guidance on when to choose it over siblings like codeinspectus_rescan, codeinspectus_compliance_report, or codeinspectus_generate_sbom. There are no exclusions, prerequisites, or alternative-selection hints, so an agent must infer when this tool is the right one.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codeinspectus_explain_findingA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint, idempotentHint, and destructiveHint, so the description doesn't need to cover safety. It adds value by clarifying that it returns a full remediation plan and references, which is beyond the annotations. No contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that efficiently covers purpose, content, and context without redundancy. It is front-loaded with the main action and includes all key elements.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema present, the description need not explain return values. The tool is clear for a read-only operation, and the parameters are well-defined in schema. It is complete enough for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with descriptions for both parameters. The description adds minor value by explaining the output context ('from a prior scan') and giving example finding ID (CI-0007) in the schema, but it doesn't elaborate on the structure or format beyond that.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns a deep explanation and remediation plan for a single finding, distinguishing it from scan/rescan tools. However, it does not explicitly contrast with listing findings or compliance report, but the focus on 'single finding id' and 'remediation' sets it apart.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies it is used after a prior scan to get details on a specific finding, but it does not provide explicit when-not-to-use guidance or mention alternatives like compliance_report for broader context. It relies on the sibling list to imply context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codeinspectus_list_rulesA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false, so the safety profile is well covered. The description adds useful specificity about the returned inventory categories, but discloses no additional behavioral caveats such as rate limits, freshness semantics, or output size. This is adequate given the strong annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single information-dense sentence that enumerates the full scope without fluff. It could be split or trimmed slightly, but every listed item adds meaning and nothing is repeated from the schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With one optional parameter, a full output schema, and strong read-only/idempotent annotations, the description is complete for this inventory-query tool. It covers all major categories of returned data, and the schema handles engine filtering, leaving no significant gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%: the only parameter, 'engine', has a descriptive enum and clear filtering semantics. The description does not mention this filter, but the schema carries the parameter meaning entirely, so the baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') with an explicit resource scope: active detectors, engine versions, database freshness, advisory provenance, AI-code rules, and detector packs. This clearly distinguishes it from sibling tools like scan, rescan, setup, and compliance_report.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is the inventory/inspection tool by emphasizing 'active' and 'currently shipped,' but it does not explicitly state when to use it versus alternatives or provide any exclusions. Usage context is present but only by implication.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codeinspectus_compliance_reportA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint, idempotentHint, and non-destructive. The description adds valuable context: it is 'code-level evidence only' and not an audit, and describes the output format ('X of N code-visible controls have findings' with explicit denominator). This goes beyond annotations to set correct expectations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core action and resource, and includes a clear caveat. No extraneous information; every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema, complete input schema, and annotations, the description sufficiently covers the tool's purpose, input requirements, output format hint, and limitations. It does not need to explain return values since output schema exists, and for a read-only reporting tool, this is complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema covers 100% of parameters with descriptions, so the burden on the description is low. The description mentions the frameworks in parentheses, mirroring the enum, and explains the per-framework view, which aligns with the optional framework parameter. This adds minimal extra meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool produces a 'per-framework code-level control-coverage view for a prior scan,' specifies the supported frameworks, and explicitly distinguishes itself from a compliance audit. The verb 'produce' and resource 'code-level control-coverage view' are specific. Sibling tools like codeinspectus_explain_finding have different purposes, so this tool is well-differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description notes this tool is for a 'prior scan,' implying it should be used after a scan. It also warns that it is NOT a compliance audit or attestation, guiding against misuse. However, it does not explicitly state when to use this tool over specific siblings like codeinspectus_explain_finding, leaving some ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codeinspectus_rescanA
  4.3/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description adds meaningful behavioral detail beyond the annotations: it reports resolved/remaining/newly introduced findings, fresh technology and native-pack coverage, and fail-closed states for repository-trust artifacts. It also reassures that the tool never writes to code or repo, matching the readOnlyHint and destructiveHint annotations without contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the core operation. Each sentence adds distinct value: the action, the diff output, the trust-artifact nuance, the intended use, and the read-only guarantee. There is no wasted content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a rescan tool with an output schema and read-only annotations, the description covers the primary workflow, the diff semantics, and the safety profile. The main gap is that three optional parameters are not explained, though their names and enums are reasonably self-descriptive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 40%, so the description should compensate for undocumented parameters. It does not explain scanners, max_findings, or severity_threshold, and its mention of prior_scan_id only restates what the schema already documents. The optional parameters remain underspecified.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with a specific verb and resource: 'Re-run a scan... and diff against a prior scan_id.' It clearly distinguishes this from the sibling codeinspectus_scan by emphasizing the post-fix diff workflow rather than an initial scan.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says 'Use this to verify approved fixes,' which gives a clear when-to-use trigger. It does not explicitly name an alternative or state when not to use it, but the purpose and sibling context make the intended use fairly obvious.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codeinspectus_generate_sbomA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations, the description discloses that it writes a file, returns a file location plus component count, uses Trivy with a first-party offline Pub lockfile inventory, and falls back to native Pub. It also explicitly states 'Offline,' adding meaningful behavioral context. There is no contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, each carrying significant information: what is produced, how, under which conditions, fallback behavior, and output semantics. There is no filler or redundant restatement of the tool name or schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the moderate complexity and the presence of a full output schema, the description sufficiently covers the tool's behavior, output, offline nature, and fallback paths. It provides enough context for an agent to select and invoke it appropriately without ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already documents all three parameters with clear descriptions and default values, so the description adds only minor reinforcement like 'chosen output path.' It does not introduce new meaning beyond the schema's existing high coverage, resulting in a baseline-level score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Generate'), the exact deliverable ('CycloneDX or SPDX SBOM'), and the target ('target project'), distinguishing it clearly from sibling tools like scanning or reporting. It also communicates the method and file output, making the tool's scope immediately obvious.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly implies when to use this tool: whenever an SBOM is needed for a target project, with explicit offline behavior and fallback logic. It does not list alternatives or exclusion conditions, but no sibling tool appears to duplicate SBOM generation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• codeinspectus_setupA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint=false, destructiveHint=false, idempotentHint=true, and openWorldHint=true. The description adds valuable context beyond annotations: it specifies that install writes only to ~/.codeinspectus, verifies immutable pins/publisher provenance, and never modifies the target repository. It also clarifies that plan is offline. This is strong behavioral disclosure, though it could mention network access requirements or failure modes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary purpose, and every clause adds value. It efficiently covers the key behaviors without redundancy. The structure is clear and scannable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (3 params, 3 actions, 3 components) and the presence of an output schema, the description is quite complete. It covers the main behaviors, safety guarantees, and offline/online distinction. It could be slightly more explicit about what 'plan' returns or how decline works, but the output schema likely covers return values. The description is sufficient for an agent to select and invoke this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all three parameters well. The description adds context by explaining the action parameter's behavior (plan is offline, install requires confirm_downloads=true, decline saves choice) and the components parameter (default all three). This goes beyond the schema's basic descriptions, though the schema already covers the essentials.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: inspect external-engine health, plan or install specific components (Opengrep/Gitleaks/Trivy) after confirmation, and save declined choices. It uses specific verbs (inspect, plan, install, save) and names the resources, distinguishing it from sibling tools like scanning or reporting.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use this tool (for external engine setup) and provides clear behavioral guidance: 'Plan is offline', 'Install writes only to ~/.codeinspectus', 'never modifies the target repository'. It also implies when not to use it (not for scanning or rule management, which are siblings). The action parameter description further clarifies when each action is appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.