Server Quality Checklist

Tool Scores

• kb_list_documentsB
  3.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries the burden of behavioral disclosure. It clarifies that bodies and embeddings are not included, which is a key behavioral trait. However, it does not mention other aspects like pagination, ordering, or whether filtering is exact or partial. It is adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that immediately conveys the core purpose and scope. Every word contributes value, and there is no redundancy or filler. It is perfectly concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters (0 required) and a tool with multiple filter options, the description is too brief. It does not explain how filters combine, the meaning of each field, or what the output schema contains. An agent cannot fully judge whether this tool meets a specific filtering need without additional inference.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0% and the description provides no information about the 5 parameters (limit, domain, status, service, document_type). The description must compensate due to low coverage but does not, leaving the agent to rely solely on parameter names and types, which may be insufficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it lists filtered document metadata and explicitly excludes bodies and embeddings. This distinguishes it from siblings like kb_search (full text) and kb_get_document (full document). The verb 'list' combined with the resource 'document metadata' is specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives such as kb_search or kb_get_document. The description implies use for metadata, but does not mention exclusions or specific scenarios. An agent would have to infer usage from the purpose.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kb_statsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description must cover behavioral traits. It discloses the output elements but does not mention performance, required permissions, or side effects. As a stat retrieval tool, it is likely safe, but lacks explicit reassurance.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with key outputs. Could be improved by structuring or clarifying terms like 'identity', but overall efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters and an output schema, the description is nearly complete for its simplicity. However, it does not address how this tool relates to sibling tools, leaving some contextual gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (no parameters). Baseline score of 3 applies as the description adds no parameter info, but none is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it returns index counts, identity, timestamps, and resolved local directories, indicating a read-only stats tool. However, it does not differentiate from siblings like kb_search, kb_get_document, kb_list_documents, and 'identity' is vague.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. It is implied for getting overall knowledge base stats, but no explicit when-to-use or when-not-to-use criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kb_get_documentA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It lacks details on permissions, rate limits, side effects, or what 'normalized' means. As a retrieval tool, it likely has no destructive side effects, but this is not confirmed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no wasted words. Every part adds value, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one parameter and an output schema, the description covers purpose and usage context. However, it lacks behavioral transparency and parameter details, leaving gaps in completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description mentions document_id comes from kb_search but does not explain its format, constraints, or allowed values. With 0% schema description coverage, more detail is needed to compensate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('return'), the resource ('one complete normalized document'), and the context ('after kb_search identifies its document_id'), effectively distinguishing it from sibling tools like kb_search and kb_list_documents.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implicitly tells when to use this tool (after kb_search) and implies it is not for searching or listing. However, it does not explicitly state when not to use or mention alternatives beyond the sibling names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kb_searchA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses that results are fragments, not complete documents, and mentions citing sources. However, it does not specify behavior like how results are ranked, pagination, or any side effects. Still, for a search tool, this is reasonably transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each serving a distinct purpose: what it does, how to use it, and when to use an alternative. No wasted words, well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (9 parameters, no schema descriptions, no annotations), the description covers high-level purpose and guidance but fails to document parameters. Output schema exists but does not compensate for missing parameter semantics. Completeness is moderate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 9 parameters with 0% description coverage. Description only implicitly mentions 'query' as the search term and does not explain any other parameters (top_k, domain, status, service, authority, min_score, source_type, document_type). This leaves the agent guessing about their meaning and usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states it searches corporate knowledge, lists concrete use cases (architectural analysis, changes across services), and mentions specific content types (business rules, ADRs, APIs, events, runbooks). It also distinguishes from sibling tool kb_get_document for complete documents.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear instructions: use before architectural analysis, cite source_path or source_url, and notes that a fragment is evidence, not the only source of truth. Explicitly suggests using kb_get_document when the complete document is needed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.