Server Quality Checklist

Tool Scores

• get_operationA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses an important trait: the operation is not invoked, indicating a read-only, side-effect-free behavior. However, it does not mention error behavior (e.g., what happens on no match) or return format details, so transparency is partial. The explicit 'does not invoke' is valuable but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences deliver the core purpose, a constraint, and a crucial non-invocation note — no filler. The main action is front-loaded, and every clause earns its place. It is efficiently structured for quick comprehension.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  As a retrieval tool with no output schema, the description should at least indicate what structure the 'operation' takes or what happens when no match is found. It says it 'returns one operation' but does not describe the return object or error cases. Given the sibling tools and the absence of an output schema, the description is under-specified for an agent to know exactly what to expect or how to handle failures.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is only 33% (only 'path' has a description). The description adds meaning for 'method' and 'path' by stating they must match the published spec exactly, but it never explains 'apiId' — what it represents or how to obtain it. Thus, two of three parameters get some semantics, but the most opaque one remains undocumented, and the tool text does not fully compensate for the low schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear action ('Return one operation') and resource ('from a cataloged API'), and distinguishes itself by explicitly noting it 'describes the operation; it does not invoke it.' This sets it apart from tools that might execute API calls and makes its purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It conveys a prerequisite ('Method and path must match the published spec exactly') and a key differentiator ('does not invoke it'), but it does not explicitly compare with sibling tools like list_catalog or inspect_contract. The context of when to choose this over alternatives is implied rather than stated, so some guidance exists but lacks explicit routing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspect_contractA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full behavioral burden. It discloses that the tool returns manifests, deny-list ids, and a statement of what the server is not, indicating a read-only inspection. However, it doesn't mention any potential side effects, resource costs, or error behavior. For a simple read-only tool this is adequate but not rich.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that directly states what is returned. No filler, every word earns its place, and the key outputs are front-loaded. This is appropriately concise for the tool's simplicity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a parameterless inspection tool with no output schema, the description covers the essential return values. It doesn't explain what 'tool manifest' or 'resource manifest' mean, but that may be domain-specific. It also doesn't clarify when to choose this over siblings, but given the tool's narrow scope, it's relatively complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so there is nothing for the description to explain. The baseline for 0 parameters is 4, and the description doesn't need to compensate for any schema gaps. The tool is straightforward in that regard.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a clear verb ('Return') and specifies the resource (contract information: tool manifest, resource manifest, deny-list ids). It is specific about what is returned, but it doesn't explicitly contrast with sibling tools like list_catalog or explain_deny, leaving some ambiguity about what makes this unique.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use it (for inspecting the server's contract) but doesn't state explicit conditions or exclusions relative to the siblings. It doesn't say 'use this when you need an overview' or 'don't use this for looking up specific operations'. Context is somewhat clear but not fully articulated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• explain_denyA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations present, the description carries the full burden of behavioral disclosure. It clearly states the tool's purpose but does not mention side effects, permissions, or that it is a read-only operation. The phrase 'refused' implies it is explanatory, but the description does not explicitly say whether it executes anything or how it behaves beyond returning an explanation. This is adequate but not rich; a 3 reflects the absence of contradictions balanced against incomplete behavioral detail.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with zero redundancy. The primary purpose is front-loaded, and the usage guidance follows immediately. Every word earns its place without padding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one enum-constrained parameter and no output schema, the description provides sufficient context: it explains what it does and when to prefer it over requesting capabilities. It does not describe the return format or potential errors, but given the simplicity and the absence of an output schema, this is a minor gap. Completeness is high for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides 100% coverage: the single parameter 'capability' has an enum list and a description that fully defines it as 'A capability from the published deny-list.' The description adds no extra semantics beyond referring to a 'named capability,' which is already covered by the schema. Given high schema coverage, the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear verb-resource purpose: 'Explain why a named capability is refused.' It also distinguishes itself from asking the server to grow a shell, HTTP client, or secret reader, which positions it as a diagnostic tool rather than a capability-granting one. However, it does not explicitly name the sibling tools (list_catalog, lookup_api, etc.) or contrast with them, so some differentiation is left implicit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides explicit guidance on when to use this tool: 'Use this instead of asking the server to grow a shell, HTTP client, or secret reader.' This tells the agent that whenever it would consider requesting such capabilities, it should instead call explain_deny. It does not mention the sibling tools or give extensive exclusion criteria, but the core use case is clearly stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_catalogA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the return fields (id, owner, lifecycle, classification, operation count) and the 'No network' behavior, which is valuable context. It does not mention potential staleness or failure modes, but for a simple list these are minor omissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with zero waste. The verb and resource are front-loaded, followed by return fields and a key behavioral note. Every clause earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (no params, no output schema), and the description fully explains what it does and what it returns. Sibling tools exist but differentiation is handled by purpose clarity. Nothing essential is missing for an agent to invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so schema coverage is trivially 100%. Per calibration, a baseline of 4 applies. The description adds nothing parameter-related because none exist, which is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'List' and the resource 'static OpenAPI catalog entries published with this server', distinguishing it from siblings like lookup_api (single lookup) and get_operation (specific operation). The scope is explicit and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description notes 'No network' implying a fast, static listing, but does not explicitly say when to use this vs alternatives such as lookup_api. It offers no explicit exclusions or conditions, but the purpose is self-evident enough that an agent can infer suitability for overview retrieval.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lookup_apiA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It reveals two important traits: it rejects unknown ids (error behavior) and does not fetch URLs from the spec (avoiding network side-effects). Even without a return format specification, these disclosures cover the key operational concerns an agent would have.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences, front-loaded with the core action and resource. The second sentence delivers two high-value constraints efficiently. There is zero filler; every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple single-parameter lookup with no output schema and no annotations, the description covers the essential purpose, error handling, and a limitation. It does not describe the return structure (e.g., raw JSON vs. parsed object), but that is a minor gap given the simplicity and the explicit scope of the tool. Overall, an agent can confidently invoke it.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already documents apiId with a pattern and example, achieving 100% coverage. The description adds concrete catalog id examples (orders, identity, billing) that go beyond the schema's single example, enriching the parameter's semantic meaning and helping the agent select valid values. This justifies a score above the baseline 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Return' and the resource 'imported OpenAPI document' with a specific identifier (catalog id). It also gives concrete examples (orders, identity, billing) and implicitly differentiates from siblings like list_catalog (which lists) and get_operation (which fetches a specific operation). The purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides practical usage guidance by stating that unknown ids are rejected (so the agent knows to validate against the catalog) and that it does not fetch URLs from the spec (clarifying a limitation). It does not explicitly name alternatives, but the examples and behavior imply it is for looking up a single document by id, which is sufficient for most agents to route correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.