Server Quality Checklist

Tool Scores

• search_papersC
  2.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions searching 'across multiple sources' but does not cover critical aspects such as authentication needs, rate limits, pagination, or what the response format looks like. This leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is brief and front-loaded with the main purpose, but the 'args' section is somewhat redundant as it repeats parameter names without adding new insights. It could be more structured to avoid duplication and enhance clarity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of a search tool with no annotations and no output schema, the description is incomplete. It lacks details on result format, error handling, source specifics, and behavioral traits, making it inadequate for full contextual understanding.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful context for both parameters: 'query' is explained as 'the search query,' and 'limit' as 'the maximum number of results to return (default 10).' Since schema description coverage is 0%, this compensates well by clarifying parameter purposes beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool 'Search for papers across multiple sources,' which provides a clear verb ('Search') and resource ('papers'). However, it does not differentiate from sibling tools like 'search_by_topic' or specify what 'multiple sources' entails, making it somewhat vague in distinguishing its unique scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'search_by_topic' or 'fetch_paper_details.' The description lacks context on scenarios, prerequisites, or exclusions, leaving usage decisions unclear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fetch_paper_detailsB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool 'Get[s] detailed information,' which implies a read-only operation, but it doesn't disclose any behavioral traits such as authentication needs, rate limits, error handling, or what 'detailed information' includes. This leaves significant gaps in understanding how the tool behaves beyond its basic purpose.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with a clear purpose statement followed by a concise 'Args' section that lists parameters with brief explanations. Every sentence earns its place by providing essential information without unnecessary details, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no annotations, no output schema), the description is partially complete. It covers the purpose and parameters well, but it lacks information on behavioral aspects like what 'detailed information' entails, potential errors, or usage constraints. Without an output schema, the description should ideally hint at the return structure, but it doesn't, leaving some context gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful semantics beyond the input schema, which has 0% description coverage. It explains that 'paper_id' is a 'Paper identifier (DOI for Crossref, paper ID for Semantic Scholar)' and 'source' is a 'Source database' with options 'semantic_scholar' or 'crossref'. This clarifies the purpose and format of the parameters, compensating well for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get detailed information about a specific paper.' This specifies the verb ('Get') and resource ('paper'), making it easy to understand what the tool does. However, it doesn't explicitly differentiate from sibling tools like 'search_by_topic' or 'search_papers', which likely return lists rather than details for a specific paper.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by specifying that it's for a 'specific paper' and lists the required 'paper_id' and optional 'source' parameters. This suggests it should be used when you have a known paper identifier, but it doesn't explicitly state when to use this tool versus the sibling search tools or provide any exclusions or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_by_topicB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds useful context: the query length limit (300 characters with truncation) and the return type (formatted search results or error message). However, it lacks details on permissions, rate limits, error conditions beyond truncation, or pagination behavior, which are important for a search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and well-structured. It starts with a clear purpose statement, adds a critical behavioral note (query length limit), and then lists parameters and returns in a formatted way. Every sentence adds value without redundancy, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (4 parameters, no output schema, no annotations), the description is partially complete. It covers parameters and basic behavior but lacks output details (e.g., result format beyond 'formatted'), error handling specifics, and differentiation from siblings. For a search tool, this leaves gaps in guiding the agent effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds significant meaning beyond the input schema, which has 0% description coverage. It explains each parameter's purpose: 'topic' as the search query with a character limit, 'year_start' and 'year_end' for date range, and 'limit' for maximum results with a default. This compensates well for the schema's lack of descriptions, though it could note that year parameters are optional integers.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search for papers by topic with optional date range.' It specifies the verb ('search'), resource ('papers'), and scope ('by topic with optional date range'), making the intent unambiguous. However, it doesn't explicitly differentiate from sibling tools like 'search_papers' or 'fetch_paper_details,' which would be needed for a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'search_papers' or 'fetch_paper_details.' It mentions optional parameters like date range and limit, but doesn't explain scenarios where this tool is preferred over siblings or any prerequisites for usage. This leaves the agent without context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.