Server Quality Checklist

Tool Scores

• get_capabilityA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true, idempotentHint=true, and destructiveHint=false. The description adds meaningful behavioral context beyond that: every capability is returned with its proving evidence and bounding limitation, and the tool cannot return one without the other. This is useful but does not describe not-found behavior or other edge cases.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded, with the first sentence stating purpose and lookup modes. The second sentence adds a valuable invariant, though the dash clause 'this tool cannot return one without the other' slightly restates the preceding point rather than adding new information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is adequate for a simple read-only lookup: it identifies resource, lookup modes, and a key return invariant. However, with no output schema and one undocumented parameter (`limit`), it should clarify parameter interaction and behavior when no capability is found; it also does not offer guidance relative to the sibling tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 67%; id and topic are documented in both schema and description, but the description adds little beyond what the schema already provides. The `limit` parameter is not explained semantically, and the description does not clarify whether id and topic are mutually exclusive or can be combined.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Resolve' with a clear resource ('capability from the claims ledger') and names three supported lookup paths: claim id (C-nn), standing-limitation id (L-nn), or topic. It also distinguishes this from siblings like check_job and search_docs by emphasizing the claims-ledger domain and the evidence/limitation pairing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly communicates when to use the tool by specifying lookup by id (C-nn), limitation id (L-nn), or topic. It implies topic is useful when the id is unknown, though it does not explicitly name alternatives like check_job or search_docs or provide when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_jobA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare read-only, idempotent, and non-destructive behavior. The description adds value by detailing output semantics: it returns statuses (not supported / partially supported / technically supported / validated end to end), supporting tests, and an explicit answer when not supported. This goes beyond annotations to set expectations about the response shape and edge-case handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The two-sentence description is dense with information, front-loading the core question and then detailing the output types. It is not overly verbose and conveys a lot in few words. A slight deduction for not being even more compact, but it remains appropriately sized for the complexity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only query tool with one required and one optional parameter, the description explains what the tool returns, including the status taxonomy and the explicit handling of unsupported jobs. It mentions tests as evidence, which provides completeness. There is no output schema to compensate for, so the description covers the essential aspects, though it could have mentioned pagination or limit behavior, which is minor.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema documents 'query' fully, including an example, and 'limit' with constraints. Description coverage is 50%, and the description does not elaborate on parameters, but the schema already covers the main fields. The phrase 'against the CRM jobs-to-be-done index' adds retirement domain context to the query parameter. However, 'limit' remains semantically thin, and no new parameter information is provided in the description, so it neither helps nor hurts beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Answer') and identifies a precise resource ('the CRM jobs-to-be-done index'). It clearly differentiates the tool by describing the query format ('can this framework do X?') and enumerates the kind of results (jobs with statuses, tests, explicit answer for unsupported). This distinguishes it from generic search or capability tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes the intended use clear: answering capability questions about a framework against a CRM index. However, it does not explicitly mention when to use this over siblings like get_capability or search_docs, though the specialized index and status taxonomy imply a niche. It lacks explicit exclusions or alternative guidance but provides clear contextual usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_docsA
  4.1/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses the output format (file path, nearest heading, excerpt per hit) and explicitly labels the tool as read-only and non-customer-data, matching the annotations. No contradictions or surprises.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that conveys the essential information without unnecessary fluff. It is perfectly concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description explains what the tool does, what it returns, and its read-only nature. It doesn't cover edge cases or error handling, but for a straightforward search tool, this is adequate. Minor lack of detail about the 'nearest heading' concept, but overall complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description does not add any information about the parameters 'query' and 'limit' beyond what is already in the schema. Since the schema already describes both parameters, the description adds no value to parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action ('Keyword search') and specific resource ('this framework's documentation set'), listing the included document types. It distinguishes itself from sibling tools (get_capability, check_job) by focusing on documentation search.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly states the tool is read-only and never serves customer records, implying it should be used for documentation queries. However, it doesn't provide explicit guidance on when to prefer this tool over alternatives, though the purpose is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.