Server Quality Checklist

Tool Scores

• search_codeA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden. It discloses that the search is semantic, which is useful, but does not describe other behavioral aspects like the various modes, result scoring, or return format. The scope restriction adds some context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise, front-loaded sentences with no wasted words. The purpose and a typical use case are presented efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With five parameters, no annotations, and no output schema, the description is too thin. It omits details about mode selection, filtering options, and what the response looks like, leaving the agent to guess significant parts of the tool's behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description does not explain any of the five parameters. It only implies the query is a search term, offering no additional meaning beyond the schema's basic field names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs semantic search over code examples in the docs, using a specific verb and resource. It distinguishes itself from sibling search_docs by the 'restricted to code examples' scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides a concrete use case ('show me a script that does X') that signals when this tool is appropriate. It does not explicitly exclude search_docs, but the restriction to code examples implies a clear boundary.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_chunkA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool returns the 'full content' of a chunk, implying a read operation. However, it does not mention error behavior, permissions, or any edge cases. For a simple fetch tool, this is minimal but acceptable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loads the purpose, and contains no redundant information. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 1-parameter tool with no output schema, the description is sufficiently complete. It explains the action, the usage context, and indicates the return value ('full content'). It could mention not-found behavior, but the overall context is clear for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides no description for chunk_id (0% coverage), so the description must compensate. It adds context by stating this ID corresponds to a 'specific hit' from a search_docs call, which helps the agent understand where the ID comes from. It does not specify format or constraints, but with only one parameter, this is adequate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Fetch' and specifies the resource 'a single chunk by its ID', clearly distinguishing it from sibling tools like search_docs and search_code. It also states the purpose of retrieving full content for a specific hit, which is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly provides a usage context: 'Use after a search_docs call to get the full content of a specific hit.' This clearly indicates when to use the tool. It does not explicitly mention when not to use it or compare with alternatives, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_docsA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of disclosing behavior. It states the return format ('top-k chunks with file paths, headings, and similarity scores') and the semantic search nature. For a read-only search tool, this is sufficient, though it omits potential edge cases or permission requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three short, information-dense sentences with no filler. It front-loads the core purpose and then adds usage and output details efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's purpose, typical usage, and return format, which is adequate for a straightforward search tool. It does not elaborate on parameter variations, but the schema provides the essential details, making the overall context reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage, requiring the description to compensate. It only indirectly references top_k via 'top-k chunks' and does not explain the meaning or usage of mode, min_score, publication, or product_area, which are essential for correct invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs semantic search over documentation, specifying the exact scope. It also differentiates itself from the sibling search_code by targeting documentation rather than code, and by highlighting 'conceptual or how do I' queries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Best for conceptual or how do I queries' provides a clear usage context, guiding when to prefer this tool. However, it does not explicitly mention alternatives or when not to use it, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_bundle_infoA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It explicitly lists the return fields, making the output transparent. It implies a read-only operation via 'Return' and gives a use case, though it doesn't explicitly state side-effect freedom or auth requirements, which are negligible for a metadata retrieval tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary action ('Return the bundle manifest'), and every word contributes value. No filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple metadata retrieval tool with no parameters and no output schema, the description fully enumerates the return fields and the intended use case. It is complete and self-contained.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has no parameters, so the baseline of 4 applies. The description correctly avoids unnecessary parameter details, as there is nothing to explain.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns the bundle manifest with specific fields (source repo/branch/SHA, build date, chunk count, embedding model). This distinguishes it from sibling tools like search_docs and get_chunk, which focus on content retrieval rather than metadata.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  'Use to cite freshness' provides explicit guidance on when to invoke this tool. The contrast with search/retrieval siblings is implicit but clear. It doesn't explicitly state 'when not to use' or name alternatives, which is a minor gap.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.