Server Quality Checklist

Tool Scores

• list_dated_files_toolA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. The description does state 'Read-only,' which is a key behavioral trait. However, it does not describe edge-case behavior such as what happens when a folder contains no dated filenames, or whether the listing follows any particular ordering or hidden-file handling beyond the output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and front-loaded: the primary function is stated in the first sentence. The second sentence, 'Helps an agent decide what to ask next,' adds light context but is somewhat filler-like. Overall, it is concise with no repetition of schema fields.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple, has one parameter, and an output schema exists, so return value details are not required. The description gives enough information to understand the basic call: a folder is required and the result will list reporting periods. However, it does not explain how to choose this over sibling tools, which leaves a notable completeness gap for an agent navigating multiple related tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must add meaning beyond the schema. The description confirms that the 'folder' parameter is the source of dated filenames and that reporting periods are derived from them. However, it does not provide details about expected path format or accepted folder specifiers, though the single parameter is relatively self-explanatory.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'List which reporting periods a folder holds from dated filenames.' This clearly identifies the tool's function and differentiates it from sibling tools like check_coverage_tool and check_staleness_tool, which are about validation rather than inventory.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Helps an agent decide what to ask next' implies use as an exploratory step, but the description does not explicitly say when to use this tool versus the sibling check tools. No exclusions or alternative conditions are provided, so guidance is only implied.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_coverage_toolA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description explicitly labels the operation as read-only, which is an important safety disclosure. It does not detail how 'present' is determined or what the exact output is, but the presence of an output schema reduces that burden.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: the core purpose appears first, followed by the read-only note and parameter usage. Every sentence adds value, and the example period ranges are useful without adding clutter.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only check with one required and one optional parameter, the description gives enough information for basic invocation: purpose, inputs, and safety behavior. It lacks explicit guidance on choosing between sibling tools and a precise definition of 'folder span', but the output schema presumably covers return details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Since the schema provides 0% description coverage, the description compensates by naming both parameters: folder is the folder to inspect, and period_range is an optional range with useful examples like 'January 2024 to December 2025' and 'last 12 months'. It could add folder path format or the meaning of a null period_range, but the essential semantics are covered.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear action: verify that every month in a folder's span is present, optionally restricted to a period range. This distinguishes it from list_dated_files_tool and check_set_coverage_tool by focusing on month-level coverage, though it does not explicitly name or contrast those siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool through its purpose and gives usage context for the optional period_range with concrete examples. However, it does not explicitly say when to prefer this tool over check_staleness_tool or check_set_coverage_tool, nor does it mention any exclusions or preconditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_staleness_toolA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It explicitly states that the tool is read-only and discloses the UNCHECKABLE return condition for missing recorded facts. It also clarifies that it does not perform fuzzy searching. This provides meaningful behavioral context, though it does not detail all possible outputs or error conditions, which are partly covered by the output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise—three sentences. It front-loads the core purpose in the first sentence, then adds essential constraints and return behavior. No wasted words; every sentence contributes meaningful information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (though not shown) and no annotations, the description covers the essential aspects: purpose, read-only nature, required paths, and a special return condition. It does not explicitly discuss error handling for missing files or permissions, but the read-only designation and simple scope make this acceptable. It is largely complete for a staleness-check tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, so the tool description must compensate. It does add value by explaining that both paths (document and source) are required and that recorded_facts affect the UNCHECKABLE return. However, it does not elaborate on the role of each parameter beyond their names (e.g., what a 'source file' is versus 'document'), leaving some semantics to be inferred.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action (check) and a clear subject (whether a document's figures still match a source file). It also distinguishes itself by noting that it does not search for a plausible match, which helps differentiate it from sibling tools like check_coverage_tool. The purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (when you need to verify staleness against a named source) and provides a key constraint (both paths are required). However, it does not explicitly mention alternative tools or when not to use it, leaving some ambiguity against siblings. The guidance is implicit rather than direct.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_set_coverage_toolA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does so well: it states 'Read-only, and touches no filesystem,' explains that `expected` is never inferred, and discloses the behavior for unexpected items ('deliberately earns no credit against the denominator'). This gives an agent a trustworthy model of side effects and result semantics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description front-loads the core purpose and usage, then uses later paragraphs for parameter semantics and output behavior. Each section earns its place; the examples are compact and collectively clarify the tool's scope without unnecessary fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read-only, pure computational tool with an output schema and no side effects, the description is complete: it covers safety, parameter meanings, returned fields, and the special treatment of unexpected items. No critical information an agent would need to call it correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, and it does for most parameters: `expected` is defined as the caller's declaration, `scope` names items for the sentence, `where` names where they were looked for, and `derivation` records how the expected set was arrived at. `found` is only implied as 'what was actually read' rather than explicitly named as the second required list, leaving a small gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a precise verb-resource pairing: 'Check what a task required against what was actually read, over any two sets of keys.' It also distinguishes itself from dated-file-oriented siblings by saying 'when the thing you must account for is not dated files in a folder' and provides concrete example pairings, so an agent can tell this apart from related tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage context: 'Use this when the thing you must account for is not dated files in a folder,' followed by several illustrative scenarios such as documents vs. retrieved chunks and required controls vs. controls with evidence. It does not name alternatives or state explicit when-not-to-use cases, but the guidance is clear enough to route an agent correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.