Server Quality Checklist

Tool Scores

• search_docsA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and idempotentHint=true, so the safety profile is known. The description adds behavioral details such as BM25 ranking for relevance and the return structure (list of results with fields), going beyond what annotations provide. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear opening statement, usage guidance, and a formatted Args/Returns section. It is reasonably concise, though the Args section duplicates schema descriptions to some extent, making it slightly longer than necessary.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, usage context, ranking algorithm, parameter details, and return fields, making it sufficiently complete for a search tool. It could mention edge cases (e.g., behavior when no results) but given the output schema exists and annotations are strong, it is adequately complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the description essentially repeats the parameter details (query, limit, source) without adding new meaning. Baseline for high coverage is 3; description does not enhance understanding beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches documentation and returns relevant passages with source URLs, distinguishing it from siblings like get_doc and get_doc_excerpt by focusing on general search rather than retrieving specific content. It uses a specific verb ('search') and resource ('documentation').

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit guidance on when to use the tool ('when you need to find information about specific topics, APIs, or concepts'), but does not mention exclusions or alternatives. Clear context but no explicit when-not or alternative comparisons.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_doc_excerptA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint and idempotentHint, so the safety profile is covered. The description adds behavioral context beyond annotations by mentioning the BM25 relevance algorithm and the return structure (list of excerpts with content, position, and relevance score). It does not discuss edge cases like no relevant sections, but the added context is meaningful.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: purpose, usage, args, and returns are clearly separated. The purpose and usage sentences are front-loaded and efficient. However, the Args and Returns sections largely duplicate schema information, making the description slightly longer than necessary for an AI agent that already has structured data.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, the description doesn't need to detail return values, but it does anyway. It provides purpose, usage guidance, parameter defaults, and a clear return summary. The tool is simple and read-only, and the description is complete enough for an agent to select and invoke it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% and includes descriptions and defaults for all four parameters. The description's Args section repeats the schema information without adding new semantic meaning, so it does not compensate beyond the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Get'), resource ('relevant excerpts from a large document'), and scope ('matching a query'). It also clearly distinguishes itself from get_doc by explicitly recommending 'Use this instead of get_doc for large documents'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: 'Use this instead of get_doc for large documents.' This names an alternative tool and the condition for using it. It implies when not to use it (when the document is not large, use get_doc).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_sourcesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint and idempotentHint, so the description need not restate safety. It adds value by specifying the return fields (name, url, doc_count, last_updated) and the relationship to search_docs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four short sentences cover purpose, use case, sibling relationship, and return format without redundancy. Front-loaded with the core action.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read-only list operation, the description is complete: it states what is listed, how to use it, what it returns, and how it relates to search_docs. Output schema exists, so no need to detail every field.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters and the empty input schema fully covers them, meeting the baseline of 4. The description's mention that sources can be used to filter search_docs indirectly clarifies parameter context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'List all configured documentation sources with their statistics,' a specific verb and resource. It clearly distinguishes from siblings by focusing on enumeration and discovery, and explicitly links source names to search_docs filtering.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  'Use this to discover what documentation sources are available for searching' provides direct usage context. It also notes the names can filter search_docs results, but does not explicitly contrast with refresh_sources or other alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• refresh_sourcesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that it fetches from 'all configured llms.txt URLs' and 'updates the local index,' which adds behavioral detail beyond the annotations (readOnlyHint=false, idempotentHint=false, destructiveHint=false). It also describes the return dictionary including counts and errors. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three concise sentences with the main verb phrase front-loaded. It efficiently conveys purpose, mechanism, and return value without redundant filler. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a no-parameter trigger tool with an output schema, the description is complete. It covers the purpose, the scope ('all documentation sources'), the mechanism ('fetches from llms.txt URLs'), the side effect ('updates the local index'), and the return structure. No significant gaps remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are zero parameters, and the schema coverage is 100% (empty properties). The description correctly omits any parameter details. The baseline for 0 params is 4, and the description adds no unnecessary param info, making it appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Manually trigger a refresh of all documentation sources.' It distinguishes from siblings (search_docs, get_doc, get_doc_excerpt, list_sources) by clearly indicating a mutating refresh operation rather than a read or query operation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description states 'Use this when you need the latest documentation content,' providing clear context for when to invoke the tool. It does not explicitly list exclusions or alternatives, but the sibling tools are clearly different in purpose, and the usage guidance is sufficient for straightforward selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_docA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnlyHint=true and idempotentHint=true, establishing the safe read-only nature. The description adds valuable behavioral context: pagination mechanics, the 50KB threshold, response includes has_more, and metadata fields (offset, length, total_length, has_more). It does not describe error behavior or rate limits, but the annotations cover the primary safety aspects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: opening sentence states purpose, followed by concise usage guidance, a clear Args section, and a Returns section. Every sentence contributes necessary information without redundancy, and key details are front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has moderate complexity with pagination. The description covers pagination triggers, parameter semantics, response format, and an alternative tool. It leaves no obvious gaps for a read-only fetch operation, especially given the existing annotations and output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so baseline is 3. The description adds semantics beyond individual parameter descriptions by explaining when to use offset/limit (documents >50KB) and what has_more indicates in the response. This contextual usage guidance enriches the parameter meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Get document content' with pagination support. It distinguishes from the sibling tool get_doc_excerpt by noting 'For targeted retrieval, use get_doc_excerpt instead.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use guidance: 'For documents larger than 50KB, use offset/limit to paginate' and explicitly names the alternative for targeted retrieval. This gives clear context for choosing between tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.