Server Quality Checklist

Tool Scores

• search_shelby_docsB
  3.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint=true and idempotentHint=true, indicating safe, repeatable operations. The description adds minimal behavioral context by specifying it returns 'most relevant pages', hinting at ranking or relevance scoring, but doesn't detail aspects like search algorithm, pagination, or error handling. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core action and outcome without unnecessary details. Every word earns its place, making it highly concise and well-structured for quick understanding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (search operation with 2 parameters), rich annotations (readOnlyHint, idempotentHint), and no output schema, the description is minimally adequate. It covers the basic purpose but lacks details on output format, error cases, or integration with sibling tools, leaving gaps for an AI agent to infer usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with clear descriptions for 'query' and 'limit' parameters. The description adds no additional semantic meaning beyond the schema, such as query formatting tips or result interpretation. Baseline score of 3 is appropriate as the schema adequately documents parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Search') and resource ('Shelby documentation bundle'), and indicates what it returns ('most relevant pages'). However, it doesn't explicitly differentiate from sibling tools like 'search_shelby_docs' vs 'list_shelby_doc_pages' or 'read_shelby_doc', which might have overlapping functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'list_shelby_doc_pages' or 'read_shelby_doc'. It lacks context on scenarios where searching is preferred over listing or reading specific documents, and offers no exclusions or prerequisites for usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_shelby_doc_pagesA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint=true and idempotentHint=true, so the agent knows this is a safe, repeatable read operation. The description adds minimal behavioral context beyond this (e.g., it mentions filtering but not pagination or return format). With annotations covering key safety aspects, a baseline score of 3 is appropriate for the limited additional value.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose and includes optional filtering details. Every word earns its place, with no redundancy or unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (a simple list operation), rich annotations (readOnlyHint, idempotentHint), and full schema coverage, the description is reasonably complete. It lacks output schema details, but for this type of tool, the description suffices without needing to explain return values.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with clear documentation for both parameters (prefix as an optional filter, limit with default and constraints). The description adds no additional parameter semantics beyond what the schema provides, so it meets the baseline for high coverage without extra value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('List') and resource ('Shelby documentation pages'), making the purpose immediately understandable. However, it doesn't explicitly differentiate this tool from its siblings (get_shelby_doc_chunk, read_shelby_doc, search_shelby_docs), which would be needed for a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage through the phrase 'optionally filtered by path or title text,' suggesting this is for listing with basic filtering. However, it doesn't provide explicit guidance on when to use this versus alternatives like search_shelby_docs, nor does it mention any prerequisites or exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_shelby_docA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already provide readOnlyHint=true and idempotentHint=true, indicating safe, repeatable reads. The description adds value by specifying the input types and the maxChars parameter for output truncation, but does not disclose additional behavioral traits like rate limits, error handling, or pagination. With annotations covering safety, a 3 is appropriate as the description adds some context without rich behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the purpose and key details without any wasted words. Every element (verb, resource, input types) earns its place, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, 100% schema coverage, no output schema), the description is mostly complete. It covers the purpose, input types, and implies output handling via maxChars, but lacks details on return format or error cases. With annotations providing safety context, it is adequate but could be more comprehensive for a read operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents both parameters. The description adds marginal value by listing the accepted input types for 'page' and implying output truncation with 'maxChars', but does not provide syntax or format details beyond the schema. Baseline 3 is correct when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Read' and the resource 'Shelby documentation page', specifying the exact action and target. It distinguishes from siblings by focusing on reading a specific page rather than listing, searching, or chunking, making the purpose specific and well-differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by listing the types of inputs accepted (path, title, URL, page ID, or fuzzy query), which helps guide when to use this tool. However, it does not explicitly mention when not to use it or name alternatives like 'search_shelby_docs' for broader searches, so it lacks explicit exclusions or named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_shelby_doc_chunkA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide readOnlyHint=true and idempotentHint=true, indicating safe, repeatable reads. The description adds context beyond annotations by specifying that it reads a 'specific' page and mentions the maxChars parameter for output truncation, which is useful behavioral detail not covered by annotations. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose without unnecessary words. Every part of the sentence contributes to clarity, making it appropriately sized and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (read operation), rich annotations (readOnlyHint, idempotentHint), and full schema coverage, the description is mostly complete. It lacks details on output format or error handling, which could be useful since there is no output schema, but it adequately covers the essential context for use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, with clear descriptions for both parameters. The description adds minimal value beyond the schema by implying the 'id' parameter comes from search_shelby_docs, but does not provide additional syntax or format details. Baseline 3 is appropriate as the schema handles most documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Read'), resource ('Shelby documentation page'), and scope ('by its exact chunk ID'), distinguishing it from siblings like list_shelby_doc_pages (list), read_shelby_doc (read by different identifier), and search_shelby_docs (search). It precisely communicates what the tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying 'by its exact chunk ID' and referencing 'search_shelby_docs' as the source for IDs, providing clear guidance on when to use this tool. However, it does not explicitly state when not to use it or name alternatives, such as read_shelby_doc for different access methods.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.