Server Quality Checklist

Tool Scores

• refresh_docsC
  2.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden for behavioral disclosure. It mentions the action but not side effects (e.g., overwriting index, duration, required permissions), leaving the agent uninformed about consequences.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is overly brief (one short sentence) and omits critical information (parameter meaning, usage context). While concise, it sacrifices completeness and effectiveness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one parameter and no annotation backing, the description is insufficient. It does not cover return values (despite having an output schema), prerequisites, or operational impact, leaving significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning the schema lacks parameter descriptions. The tool description does not mention the 'max_pages' parameter at all, failing to explain its purpose or effect.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's action with specific verbs ('re-crawl', 'rebuild') and resources ('source site', 'local corpus', 'index'), making it distinct from sibling tools (search/get/list) which are read-only operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives (e.g., when data is stale) or when to avoid it. The description lacks context for appropriate invocation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_docsC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions 'ranked matching pages' but does not disclose whether results include full content or snippets, how ranking works, or if any side effects exist (e.g., query logging). Important behavioral traits are omitted.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, which is concise. However, it lacks structure to efficiently convey key details. Every word is necessary, but the brevity sacrifices completeness, making it less helpful despite being short.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 2 parameters, no enums, and an output schema, the description should at least hint at expected input format and output structure. It only says 'ranked matching pages', leaving ambiguity about what the response contains. The output schema exists but is not referenced, so completeness is low.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%. The description does not explain the parameters beyond the schema, such as valid query formats, limit behavior (e.g., max value), or default ranking criteria. With no supplemental info, the agent must rely solely on the schema, which is insufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Search', the resource 'captured docs', and the outcome 'return ranked matching pages'. This distinguishes it from sibling tools: get_doc (single document), list_docs (list all), and refresh_docs (update cache).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. It does not specify that this should be used for finding relevant content, while get_doc for specific IDs, list_docs for enumeration, and refresh_docs for cache updates. The description lacks context for decision-making.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_docsB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It only states that the tool lists pages with optional filters, but does not disclose whether it is read-only, how pagination works, or what the response format is. The presence of an output schema does not excuse the description from providing behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that is efficient and front-loaded. It conveys the core purpose and key parameters without unnecessary words. Could be slightly improved with more detail, but remains appropriately concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with three optional parameters and an output schema, the description covers the basic purpose and filter options. However, it lacks details on ordering, default behavior, and total count. The output schema likely provides return value info, but the description could be more complete regarding pagination and scope.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains the 'section' and 'category' parameters by mentioning optional filtering. However, the 'limit' parameter is not described, leaving its purpose (e.g., pagination) unclear. Partial coverage but adds value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'List' and identifies the resource as 'captured pages'. It distinguishes from sibling tools: search_docs implies query-based search, get_doc retrieves a single document, refresh_docs updates data. Listing with optional filters is a distinct operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description mentions optional filtering by section or category, providing context for when to use those parameters. However, it does not explicitly state when to use this tool over alternatives like search_docs, nor does it mention any exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_docA
  3.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description is straightforward about returning page data with no hidden behaviors. No annotations are provided, but the tool is read-only in nature. It does not disclose error handling or edge cases, but the description is sufficient for a simple retrieval tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that wastes no words. It efficiently communicates the tool's purpose and output.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the low complexity (one parameter, output schema present), the description covers the return value adequately. However, the complete lack of parameter explanation leaves the definition incomplete for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage for the 'page_id' parameter. The tool description does not explain the parameter's format, expected value, or any constraints. This is a significant gap that makes it hard for an agent to know how to correctly provide the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Return') and resource ('one captured page'), and lists what is included (metadata, source URL, full markdown). It is easily distinguished from sibling tools like 'search_docs' (which searches) and 'list_docs' (which lists multiple pages).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for fetching a single page by ID, which is distinct from listing all pages or searching. However, it does not explicitly state when to use this tool versus alternatives or provide any exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.