Server Quality Checklist

Tool Scores

• explain_artifactA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint=true, idempotentHint=true, destructiveHint=false. The description adds valuable context: it is offline, requires no key, describes the return JSON structure in detail, and warns that forgeries will explain cleanly. This goes beyond annotations to inform the agent of important behaviors.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single paragraph that is well-structured and front-loaded: it states the core purpose first, then usage hints, parameter choices, return format, and a warning. Every sentence adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there is no output schema, the description fully documents the return JSON structure including fields like type, format, issuer, kid, issued_at, and payload_keys. It also covers the two parameter alternatives and warns about forgeries. The tool is relatively simple, and the description is complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% since both parameters are documented in the input schema. The description briefly restates the two input options (artifact_json or path) but adds no additional semantic meaning or constraints beyond what the schema provides. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: inspecting a signed artifact without verifying it. It uses specific verbs ('inspect') and identifies the resource ('signed artifact'), and distinguishes from sibling tools like verify_receipt by emphasizing no verification.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool (before verification to choose the right tool/key) and what it does not do (no signature check). It mentions that verification is done by verify_receipt, providing a clear alternative. It also notes that no key is required and it is read-only and offline.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_bundleA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (readOnlyHint, idempotentHint, destructiveHint), the description adds valuable behavioral details: 'Read-only; no network calls', specifies the use of embedded keys, describes the return JSON structure including error handling, and states it works offline. This fully informs the agent of what to expect.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences that efficiently convey purpose, input options, output format, and alternative tool usage. Every sentence adds value, with no redundancy or filler. It is well-structured and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has no output schema, the description fully documents the return value structure and error case. Parameters are completely covered by schema. Annotations cover safety. Sibling tool context is provided. The description is comprehensive and leaves no ambiguity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already has 100% coverage with descriptions for both parameters (path and bundle_json). The description merely restates that the bundle can be provided as raw JSON or a file path, adding no new meaning beyond the schema. Baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'verify' and the resource 'every receipt in a ScopeBlind audit bundle'. It specifies offline verification using embedded keys, and it distinguishes itself from the sibling tool 'verify_receipt' by noting when to use each.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells the agent when to use this tool (for bundles with multiple receipts) and when to use the alternative 'verify_receipt' (for single artifacts or external keys). It also notes that no network calls are needed, guiding appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• verify_receiptA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already include readOnlyHint, idempotentHint, destructiveHint. Description adds details: auto-detects formats, offline operation, specific error strings (invalid_signature, no_public_key), and key resolution logic. No contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is dense but each sentence adds value. Slightly verbose with full return JSON structure, but front-loaded with purpose. Could be slightly more concise without losing clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  No output schema, but description fully specifies return JSON fields (valid, error, type, format, kid, issuer, hash) and error conditions. Covers edge cases like unparseable input. Complete for this tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but description adds value by explaining precedence between artifact_json and path, and clarifying that public_key_hex is optional with fallback behavior. This goes beyond the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states 'Verify the Ed25519 signature of a single signed artifact' and lists specific artifact types. It clearly distinguishes from siblings like verify_bundle and explain_artifact.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides guidance on when to use artifact_json vs path, explains optional public_key_hex, and describes behavior when no key is available. However, it does not explicitly mention when to use alternatives like verify_bundle for multiple artifacts.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• self_testA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses read-only, deterministic behavior, no network calls, and not contacting ScopeBlind servers, going beyond annotations (readOnlyHint, idempotentHint) with specific operational details and return format.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences efficiently convey purpose, constraints, and call order; front-loaded with action and directly useful for agent decision-making.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters, rich annotations, and detailed description covering behavior, output shape, and recommended usage order, it is fully complete for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters present, so baseline 4 applies; description adds no extra parameter info as none exist.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool runs a packaged offline self-test that verifies a known-good sample receipt and bundle, distinguishing from siblings like verify_receipt and verify_bundle by being a self-contained trust check.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises 'Call this first to prove the verifier works before verifying your own artifacts', providing clear when-to-use guidance and implying it's a prerequisite for sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.