Server Quality Checklist

Tool Scores

• list_docsA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, and the description only says 'list', implying a read operation. No additional behavioral traits such as permissions, rate limits, or side effects are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with two sentences for purpose and one for arguments. No unnecessary words, and the core information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool, the description covers the essential: what it lists, optional directory, and return type. The presence of an output schema reduces the need for return value details. Minor lack of info on recursion or depth.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaning to the 'directory' parameter, explaining it is an optional subdirectory relative to docs root. This goes beyond the input schema which only provides type and default.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists all available documentation files, using specific verb and resource. It distinguishes from siblings like search_docs and analyze_docs, which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. The description does not mention when to use list_docs over search_docs or analyze_docs, leaving the agent to infer usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• analyze_docsA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions returning analysis results but does not disclose behavioral traits like side effects, permissions, or whether the tool is read-only. The description adds minimal behavioral context beyond purpose.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, with a clear purpose upfront followed by structured parameter explanations. It avoids fluff, though the 'Returns' line is minimal. It could be slightly tighter but is effective.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and a moderate 3 parameters, the description covers the tool's core purpose and parameter meanings. The existence of an output schema alleviates the need to explain return values. It lacks some behavioral context but is sufficient for basic use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description must compensate. It adds meaning by explaining the 'task' parameter with examples, noting 'docs' is optional, and specifying possible values and default for 'output_format'. This is helpful beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states it analyzes documentation for specific tasks, with examples like 'find_gaps', and mentions 'foundation for agent handoffs', which clearly distinguishes it from sibling tools like ask_docs or search_docs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for analytical tasks via the task examples, but does not explicitly state when to use this tool versus alternatives or provide exclusions. Sibling tools suggest different operations, but no guidance is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_docsA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description states that the tool performs 'keyword matching' and is 'semantic search ready', which hints at behavior but remains vague. No annotations are provided, so the description carries full burden. It does not disclose limitations, side effects, or performance characteristics. The return description of 'List of matching documents with relevance scores' adds some transparency but is insufficient for a 4.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, starting with a clear one-line purpose followed by a structured parameter list and return description. It avoids redundancy with schema defaults (defaults are already in schema). The structure is well-organized, though the parameter explanations could be slightly more concise for a 5.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 4 parameters, no annotations, and no explicit output schema, the description covers the basic mechanics but lacks contextual completeness. It omits information about pagination, performance, or when to use this tool versus siblings. The return description is prose, which is acceptable. The description is adequate but leaves gaps for an agent to fully understand the tool's behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It provides a clear bullet list explaining each parameter: 'query: Search query string', 'max_results: Maximum number of results to return', 'search_content: Whether to search in document content', 'search_titles: Whether to search in document titles'. This adds meaning beyond the schema's empty descriptions, fulfilling the dimension fully.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search documentation using keyword matching (semantic search ready)'. It uses a specific verb 'search' and clearly identifies the resource 'documentation'. While it doesn't explicitly differentiate from siblings, the unique verb and the mention of 'keyword matching' set it apart from 'analyze', 'ask', 'index', and 'list'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention any prerequisites, exclusions, or when not to use it. Given sibling tools like 'ask_docs' or 'analyze_docs', explicit usage context would be valuable, but it is entirely absent.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• index_docsA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must fully disclose behavioral traits. It mentions rebuilding the index but does not clarify if the operation is destructive, requires permissions, or is long-running. The return type is vague ('Indexing statistics').

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with two sentences for the main purpose and a clear Args/Returns structure. Every sentence adds value; no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, output schema exists), the description covers the action, parameter, and return value. It mentions scope ('all documentation') but could elaborate on prerequisites or side effects. Still, it is mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description carries the full burden. It explains the single parameter 'rebuild' as 'Whether to rebuild the entire index from scratch', adding meaning beyond the schema's type and default. However, it could be more precise about the effect.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Index or re-index all documentation for improved search', which is a specific verb-resource pair. It distinguishes from sibling tools like search_docs (which searches) and list_docs (which lists).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for improving search but does not explicitly mention when to use this tool versus alternatives like search_docs or analyze_docs. No when-not or alternative names are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• ask_docsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explains that the tool returns an 'AI-generated answer with sources' and mentions the user_id parameter for rate limiting. However, it does not disclose authentication requirements, cost implications, or any side effects (e.g., logging), leaving some behavioral aspects ambiguous.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: it opens with a clear purpose sentence, then lists parameters with brief explanations, and ends with the return value. Each line serves a purpose, and there is no redundant information. This makes it easy for an AI agent to quickly parse the essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (4 parameters, 1 required) and the presence of an output schema, the description is largely sufficient. It covers the main purpose, explains each parameter, and describes the return type. It does not touch on error handling or edge cases, but for a straightforward Q&A tool, this is adequate. Sibling tools are listed, providing context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. The docstring briefly explains each parameter: question, context_docs, max_context_length, user_id. While the explanations are minimal, they add meaning beyond the raw schema types. For example, 'context_docs' is described as 'Optional list of document paths to use as context', which clarifies its role. However, more detail on constraints (e.g., valid paths) would improve this.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Answer questions about documentation using AI'. It specifies the verb 'answer', the resource 'documentation', and the method 'using AI'. This effectively distinguishes it from sibling tools like search_docs (searching) and analyze_docs (analyzing), making the unique purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by stating the tool answers questions, but it does not provide explicit guidance on when to use it versus alternatives like search_docs for keyword search or analyze_docs for deeper analysis. No when-not-to-use or exclusion criteria are mentioned, limiting the agent's ability to discriminate between tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.