Server Quality Checklist

Tool Scores

• get_index_statsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. The description only states what the tool does ('Get statistics') without explaining what kind of statistics are returned, whether this is a read-only operation, if there are rate limits, or how the data is formatted. For a tool with no annotation coverage, this leaves significant gaps in understanding its behavior and output characteristics.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's purpose without any unnecessary words. It's appropriately sized and front-loaded, making it easy to understand at a glance. Every part of the sentence earns its place by conveying essential information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that the tool has no parameters, an output schema exists (which should cover return values), and the description is concise, the description is somewhat complete for its simplicity. However, it lacks context about what 'statistics' entail and how this tool fits with its siblings, leaving room for improvement in guiding the agent on usage and output interpretation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has 0 parameters, and the schema description coverage is 100% (as there are no parameters to describe). The description doesn't need to add parameter semantics beyond what the schema provides. According to the rules, with 0 parameters, the baseline score is 4, as there's no parameter information to compensate for or enhance.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Get statistics') and the target resource ('about the indexed collection'), which provides a specific verb+resource combination. However, it doesn't differentiate this tool from its siblings (get_passage_context, search_papers, search_topic), which all appear to operate on similar data but serve different purposes. The purpose is clear but lacks sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus its siblings or alternatives. It doesn't mention prerequisites, context for usage, or exclusions. While the tool has no parameters, which might imply straightforward usage, there's no explicit instruction on when this statistical information is needed compared to other tools that search or retrieve content.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_papersA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes the return format ('List of results with passage text, context, and metadata') and mentions 'relevant passages with surrounding context,' which adds value beyond basic functionality. However, it lacks details on permissions, rate limits, or error handling that would be helpful for a search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded with the core purpose, followed by clear sections for Args and Returns. Every sentence earns its place by providing essential information without redundancy, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (5 parameters, no annotations, but with an output schema), the description is largely complete. It explains parameters thoroughly and notes the return format, though it could benefit from more behavioral context (e.g., performance hints or limitations). The output schema reduces the need to detail return values, but some gaps remain in usage guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must fully compensate. It provides detailed semantics for all 5 parameters: 'query' as a natural language search query, 'top_k' with range (1-50), 'context_chunks' with range (0-3), and 'year_min/year_max' as publication year filters. This adds significant meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Semantic search over research paper chunks' which specifies the verb (search), resource (research paper chunks), and method (semantic). It distinguishes from siblings like 'get_index_stats' (statistics), 'get_passage_context' (context retrieval), and 'search_topic' (topic-based search) by focusing on semantic search over chunks.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'search_topic' or other siblings. It mentions what the tool does but lacks explicit when/when-not instructions or prerequisites, leaving the agent to infer usage from context alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_passage_contextA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It implies this is a read operation ('Expand context'), which is useful, but lacks details on permissions, rate limits, or what happens if inputs are invalid. The description doesn't contradict annotations, but it's minimal given the lack of structured data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with the core purpose stated first, followed by usage guidance and parameter details. Every sentence earns its place by providing essential information without redundancy, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (3 parameters, no annotations, but with an output schema), the description is mostly complete. It covers purpose, usage, and parameter semantics well. However, it could benefit from more behavioral context (e.g., error handling or output format hints), though the output schema mitigates some of this need.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds significant meaning beyond the input schema, which has 0% schema description coverage. It explains each parameter: 'doc_id: Document ID from search results,' 'chunk_index: Chunk index from search results,' and 'window: Chunks before/after to include (1-5).' This clarifies the purpose and constraints of all parameters, fully compensating for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Expand context around a specific passage.' This specifies the verb ('Expand context') and resource ('a specific passage'), making it easy to understand what the tool does. However, it doesn't explicitly differentiate from sibling tools like 'search_papers' or 'search_topic' beyond mentioning it should be used 'after search_papers.'

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool: 'Use after search_papers to get more context.' This gives a specific sequence and purpose. However, it doesn't explicitly state when NOT to use it or mention alternatives like 'search_topic' or 'get_index_stats,' which could help further distinguish its role.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_topicA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses key behavioral traits: deduplication by document, scoring methodology (average and best chunk relevance), sorting by average score, and filtering by year. However, it doesn't mention rate limits, authentication needs, error conditions, or pagination behavior for large result sets.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized. It begins with the core purpose, explains the deduplication and scoring methodology, lists parameters with clear semantics, and describes the return format. Every sentence adds value with zero wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 4 parameters with 0% schema coverage and no annotations, the description does an excellent job explaining parameter semantics and tool behavior. The presence of an output schema means return values don't need explanation. However, for a search tool with scoring complexity, it could benefit from mentioning performance characteristics or limitations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It provides clear semantic meaning for all 4 parameters: 'query' as natural language topic, 'num_papers' as count of distinct papers with range, and year filters as publication year bounds. This adds substantial value beyond the bare schema, though it doesn't explain null handling for year parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Find the most relevant papers for a topic, deduplicated by document.' It specifies the verb 'find' and resource 'papers', and distinguishes from sibling 'search_papers' by emphasizing deduplication and scoring methodology (average vs best chunk).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool: for finding papers by topic with deduplication and scoring. However, it doesn't explicitly state when NOT to use it or directly compare with sibling 'search_papers', which might be an alternative for paper searches without the specific scoring/deduplication approach described here.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.