Server Quality Checklist

Tool Scores

• search_policiesA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. Describes return format (ranked excerpts with context) and cache dependency. Does not disclose read-only nature or other behavioral details like rate limits or error conditions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each adding value: first states purpose, second specifies return format, third gives prerequisite. No filler. Information is front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Output schema exists, so return details are covered. Description gives a high-level sense of output (ranked excerpts). However, missing parameter details and lack of behavioral nuance about caching status make it slightly less complete than ideal for a search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, meaning no descriptions in schema. Description only mentions 'query term or phrase' for the query parameter, but does not explain the max_results parameter or provide any additional semantics beyond the parameter names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states verb 'search' and resource 'cached ASF policy documents', distinguishing it from siblings get_policy (retrieve specific), list_policies (list all), and refresh_cache (update). It specifies ranking and excerpts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states prerequisite 'run refresh_cache first', advising when the tool is usable. Does not explicitly state when not to use or directly contrast with siblings, but the context is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_policyA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must fully disclose behavior. It implies a read-only operation but does not explain the force_refresh parameter or any caching behavior. Side effects are not mentioned, but the tool is likely safe.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no wasted words. The key point is front-loaded, and the guidance is immediately after.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple retrieval tool, the description covers the main purpose and provides a prerequisite. However, it lacks details on the force_refresh parameter and potential errors. The output schema may compensate, but the description alone is slightly incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%. The description does not explain the 'key' parameter's format or valid values beyond the example keys, and 'force_refresh' is not described. This is insufficient for parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves the full text of a specific ASF policy document. It distinguishes from siblings by referencing list_policies for discovering valid keys.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly directs users to use list_policies to find valid policy keys, providing a clear prerequisite and alternative. However, it does not mention when to use search_policies or refresh_cache instead.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_policiesA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It states the tool lists all policies organized by section, which is transparent enough for a simple read operation. However, it does not disclose potential pagination, rate limits, or data volume, which could be relevant for an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with no filler. Every word adds value: 'list all available' sets scope, 'ASF policy documents' identifies the resource, 'organised by section' provides structure. It is optimally concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters and the presence of an output schema, the description is complete. It tells the agent what the tool does (list all policies by section) and nothing more is needed for correct invocation. Sibling tools provide additional context for differentiation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, and the input schema coverage is 100% (empty). According to guidelines, the baseline score is 4 when no parameters exist. The description adds no param info because none are needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'List' and clearly identifies the resource as 'all available ASF policy documents'. It also adds organization detail 'by section', which differentiates it from siblings like 'get_policy' (single document) and 'search_policies' (filtered search).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (when you need all policies organized by section) but provides no explicit guidance on when not to use it or alternatives. Given sibling names, context is inferred but not stated, so the score is neutral.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refresh_cacheA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description mentions parallel fetching and cache bypass, which are key traits. But without annotations, it should disclose potential costs (external fetches) and safety (non-destructive but mutates cache). Currently lacks these.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no redundancy. The first sentence states the main action, the second explains the parameter behavior. Efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With an output schema and a single optional parameter, the description covers the essential information for invocation. Minor gap: doesn't mention the effect on other tools' data (e.g., get_policy will now show fresh data).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter 'keys' has no schema description, but the description clarifies its meaning: omit to refresh all, provide to refresh specific. This adds value beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action (re-fetch policy documents via parallel) and resource (ASF website), and it distinguishes from sibling tools that read the cached policies.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explains when to use the tool (to bypass cache and force refresh) and how to scope the refresh (omit keys for all, provide keys for specific). However, no explicit 'when not to use' or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.