Server Quality Checklist

Tool Scores

• fetch_recordA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It does add one valuable behavioral detail: a missing record is reported via 'found=false' and is not treated as an error. This is useful context. However, it does not state that the operation is read-only, nor does it explain the simulate_outage parameter's behavior. The description provides some behavioral clarity but leaves notable gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, direct, and front-loaded with the primary action. It avoids verbosity and includes only the most essential information. The key behavioral note about 'found=false' is presented efficiently. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple fetch tool, the description covers the main action and the most important edge case (absent record). However, it omits any explanation of the 'simulate_outage' parameter, which is part of the tool's interface. While an output schema exists and relieves the description of detailing return values, the parameter gap remains. The description is adequate for basic usage but incomplete for full correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It implicitly clarifies that 'key' is the lookup identifier, but it offers no additional detail about the parameter's format or acceptable values. The 'simulate_outage' parameter is entirely unexplained—an agent cannot infer its purpose or effect from the description. Given the low coverage, this is a significant shortfall.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the primary action: 'Look up a record by key.' This is a specific verb (look up) with a clear resource (record) and method (by key). It is immediately distinct from sibling tools like summarise_lengths and slow_operation, as those suggest different operations. No ambiguity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (when you need to retrieve a record by its key) but provides no explicit guidance on alternatives or exclusion conditions. It does not mention any context where this tool should be avoided or mention sibling tools. This qualifies as implied usage, which meets the minimum viable threshold.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• slow_operationA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the full burden of behavioral disclosure. It states the core behavior (sleeping) and the intended effect (timeout demonstration), but does not mention the adjustable 'seconds' parameter, what happens after the sleep (return value or error), or any side effects. This is partially transparent but incomplete for a tool with no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, short sentence that front-loads the core action ('Sleeps') and purpose. There is no wasted wording, making it highly efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one optional parameter and an output schema, the description is fairly complete. It conveys the tool's purpose and behavior, and the output schema covers return values. However, it could explicitly mention that the sleep duration is configurable via the 'seconds' parameter, which is currently only in the schema. Overall, it is adequate for a demo tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has one parameter 'seconds' with default 60 and 0% description coverage. The description does not mention this parameter at all, so it fails to connect the sleep duration to the parameter. The parameter name is somewhat self-explanatory, but the description provides no added meaning, leaving the agent to guess how the parameter influences the operation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Sleeps' and the resource 'server's timeout behaviour', making the tool's purpose unmistakable. It distinguishes itself from siblings 'summarise_lengths' and 'fetch_record' which clearly serve different functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for demonstrating timeout behaviour, which gives context, but it does not explicitly mention when to use this tool versus alternatives. No exclusions or conditions are provided, leaving the agent to infer that it is for testing timeouts only.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• summarise_lengthsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the behavior accurately but does not discuss edge cases (e.g., empty list, null elements) or any side effects. For a pure computation tool, this is adequate but minimal disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence with no wasted words. The core functionality is stated immediately, making it quick for an agent to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple, and an output schema exists, so the description does not need to explain return values. Combined with the clear single parameter and the presence of the output schema, the description is complete for an agent to invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for the undocumented 'items' parameter. It clarifies that the parameter is a list of strings, which aligns with the schema. However, it does not add details about constraints (e.g., whether null is allowed) or the order of results. The description adds minimal value beyond the schema's type declaration.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns per-item character counts and a total for a list of strings, specifying a concrete verb and resource. It does not explicitly differentiate from siblings, but the siblings (fetch_record, slow_operation) are obviously distinct in purpose, so the lack of explicit differentiation isn't a major gap.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The purpose implies when to use it (whenever you need length summaries), but no explicit guidance is given about when not to use it or which alternatives to prefer. Since the alternatives are clearly unrelated, the implied usage is sufficient, though not explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.