Server Quality Checklist

Tool Scores

• list_pagesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare read-only and idempotent. The description adds valuable behavioral context: it is cheap, behavior changes with source size (descriptions vs paths only), and explains the cost reason. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear opening, detailed behavior explanation, param usage, and source enumeration. Every sentence adds value, though it could be slightly tighter without losing information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, the description provides sufficient context about return modes (small vs large). It covers both parameters and lists sources. Sibling tools are separate, but the tool's purpose is complete for its scope.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 50%, but the description adds meaning to the undocumented prefix parameter: 'Pass prefix to narrow it and get the descriptions back.' It also lists its values. For source, it repeats the enum but adds context. This compensates well for schema gaps.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'List a source's pages', specifying both verb and resource. It distinguishes from siblings like list_sources (different resource) and search/read tools, and adds behavior details (cheap map) that separate it from more expensive operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains when to use the tool ('cheap map of what exists') and describes behavior for small vs large sources, implying it's for quick overviews. It does not explicitly mention when not to use it or compare with siblings, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_sourcesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare the tool is read-only and idempotent. Description adds useful context about the output (page counts and freshness) beyond what annotations provide.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, 14 words, conveys essential information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters, clear annotations, and existence of output schema, the description fully covers what the tool does.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters, so schema coverage is 100%. Baseline of 4 applies as description does not need to add parameter information.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it lists documentation sets with page counts and freshness, distinguishing from siblings like grep_docs and read_doc which operate on content rather than metadata.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implied usage: when one needs an overview of available documentation sets. No explicit guidance on when not to use or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_docA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only, idempotent, non-destructive. Description adds behavior: large pages return outline, tables come in parts with header row, part parameter for pagination, and Related pages footer. No contradiction.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Well-structured into paragraphs, front-loaded with core action. Each sentence adds value: section, part, table handling, cross-references. No waste.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Has output schema (not shown) so return format likely covered. Description addresses purpose, param usage, edge cases (large pages, tables), cross-references, and sources. Complete given complexity and annotations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is low (25%). Description explains path (from search_docs), section (heading or anchor), part (for large tables), source (listed). Also explains default behavior (outline if no section). Adds essential meaning beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clear verb 'Read' and resource 'documentation page or one section of it'. Distinguishes from siblings like search_docs (returns paths) and grep_docs. Explains optional section and part for large pages.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides usage context: when to use section (large pages), how to handle huge tables (part), and following cross-references. Lists indexed sources. Could explicitly say when not to use, but implies alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• grep_docsA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide safety hints. Description adds behavioral traits: returns raw matching lines, costs more (~1.5k tokens for common term), no ranking, hits cap, and clarifies symbols are not a reason to use it. No contradictions with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is about 120 words, well-structured with bold emphasis and clear sections. Every sentence adds value: purpose, when to use, cost caveat, when not to use, parameter hint, source list. Front-loaded with key message.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given tool complexity (3 parameters, output schema exists, annotations cover safety), the description covers purpose, usage, behavior, and parameter semantics adequately. No obvious gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 33% (only 'source' described). Description adds that 'pattern' is a Python regex, and advises for 'source': 'Pass `source` unless you truly want all of them.' Also lists indexed sources in description, complementing the enum. Compensates well for low coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it does 'Regex search over the raw documentation markdown.' It distinguishes from sibling 'search_docs' by explaining it returns raw matching lines, costs more, and is a last resort. Specific verb and resource with differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance: 'Use search_docs first.' 'Come here only when search_docs missed, or when you need *every* occurrence of a literal.' It tells when not to use (for symbols, as search_docs works) and gives examples of appropriate use cases (env var, config key, flag).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_docsA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond annotations (readOnlyHint, idempotentHint), description discloses key behaviors: returns ranked snippets, matching is OR, no fuzzy matching, tokens cost ~500, symbols treated as word boundary, and that a note may indicate a word missed. Adds value beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but well-structured and front-loaded with purpose and return type. Every section earns its place (usage, behavior, query strategy, sources). Could be slightly more concise, but no wasted sentences.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity and presence of output schema, the description is highly complete: covers usage flow, query formulation, handling misses, token costs, source filtering, and cross-references sibling tools. Leaves no significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite low schema description coverage (33%), the description adds rich meaning: explains query format (English keywords, filler words dropped), limit range (1-8), and source usage (filter by product, omit only to compare or when unsure). Provides context not in schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool searches documentation and returns ranked snippets (not full pages), clearly distinguishing it from sibling tools like read_doc and grep_docs. It explicitly contrasts with grep_docs, saying not to use grep_docs just because the query contains a symbol.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance on when to use this tool first (for documented tools) and when to follow up with read_doc. Gives detailed advice on when to pass source vs omit, and strategies for re-querying when initial results don't cohere. Also notes token cost and recommends budgeting for two searches.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.