Server Quality Checklist

Tool Scores

• list_downloaded_papersB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool lists PDFs and returns file information, but it doesn't describe key behavioral traits such as whether it's read-only (implied but not explicit), performance characteristics (e.g., speed, pagination), error handling, or any side effects. The description adds minimal context beyond the basic operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured with two sentences: the first states the purpose, and the second clarifies the return value. It's front-loaded with the main action and avoids unnecessary details. However, it could be slightly more efficient by combining the sentences or omitting redundant phrasing like 'with file information' if the output schema covers it.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (0 parameters, simple list operation) and the presence of an output schema (which likely describes the return values), the description is adequate but has gaps. It covers the basic purpose and return type, but lacks usage guidelines and behavioral context, which are important for a tool with no annotations. It's minimally viable but not fully helpful for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0 parameters with 100% coverage, so no parameter documentation is needed. The description doesn't add parameter details, which is appropriate here. It gets a baseline of 4 because it doesn't need to compensate for any gaps in schema coverage, and it doesn't introduce confusion about parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'List all PDFs that have been downloaded to local storage.' It specifies the verb ('List'), resource ('PDFs'), and scope ('downloaded to local storage'). However, it doesn't explicitly differentiate from sibling tools like 'get_paper' or 'search_papers', which might also retrieve paper information but with different mechanisms or filters.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It mentions what it does but doesn't specify use cases, prerequisites, or exclusions. For example, it doesn't clarify if this is for checking local cache versus querying a database, or how it differs from 'get_paper' or 'search_papers'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_papersA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses behavioral traits such as supporting advanced query syntax with field prefixes and Boolean operators, and includes examples. However, it doesn't mention rate limits, authentication needs, pagination behavior beyond the 'start' parameter, or what the output looks like (though an output schema exists). The description adds useful context but lacks comprehensive behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded: it starts with the core purpose, then details query syntax with clear bullet points and examples. Every sentence earns its place by providing essential information without redundancy. The structure is logical and easy to follow.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (5 parameters, advanced query syntax) and rich schema (100% coverage, output schema exists), the description is mostly complete. It covers the query parameter semantics well and provides examples. Since an output schema exists, it doesn't need to explain return values. However, it could improve by mentioning behavioral aspects like rate limits or error handling, but the presence of an output schema mitigates this gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters thoroughly. The description adds value by explaining the 'query' parameter in detail with field prefixes and Boolean operators, which enhances understanding beyond the schema's basic description. However, it doesn't provide additional semantics for other parameters like 'max_results', 'start', 'sort_by', or 'sort_order'. Baseline 3 is appropriate as the schema does heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search arXiv papers with advanced query syntax.' It specifies the resource (arXiv papers) and the action (search) with a distinguishing feature (advanced query syntax). This differentiates it from sibling tools like 'download_paper', 'get_paper', and 'list_downloaded_papers', which have different functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage through examples and query syntax details, suggesting it's for searching arXiv papers with specific field prefixes and Boolean operators. However, it doesn't explicitly state when to use this tool versus alternatives like 'get_paper' (likely for retrieving a specific paper) or 'list_downloaded_papers' (likely for local files). No exclusions or prerequisites are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• download_paperA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes key behaviors: it downloads to a local directory (./.arxiv by default), handles file existence with a 'force' parameter, and returns a local file path. This covers operational aspects beyond basic functionality, though it lacks details like error handling or network behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and well-structured: two sentences that front-load the core purpose and follow with essential behavioral details. Every sentence earns its place by adding critical information without redundancy or fluff, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no annotations, but with an output schema), the description is largely complete. It explains the download action, local storage, and return value, and the output schema likely covers return details. However, it could benefit from mentioning sibling tool relationships or error cases for full completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('arxiv_id' and 'force') thoroughly. The description adds minimal value beyond the schema by implying the 'force' parameter's purpose ('even if file exists'), but doesn't provide additional syntax or format details. This meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Download PDF') and resource ('for a specific arXiv paper'), distinguishing it from siblings like 'get_paper' (likely metadata retrieval), 'list_downloaded_papers' (listing), and 'search_papers' (searching). It provides a verb+resource+scope combination that leaves no ambiguity about its function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when a PDF download is needed, but it doesn't explicitly state when to use this tool versus alternatives like 'get_paper' (which might retrieve metadata without downloading) or 'search_papers' (for finding papers). There's no guidance on prerequisites, exclusions, or comparative scenarios, leaving usage context inferred rather than clearly defined.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_paperA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool returns complete metadata and optionally downloads the PDF, adding behavioral context beyond the input schema. However, it does not cover aspects like rate limits, authentication needs, error handling, or what 'local storage' entails, leaving gaps in behavioral transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by return details and parameter guidance. Each sentence adds value: the first states the purpose, the second specifies returns, the third explains optional PDF download, and the fourth clarifies ID formats. There is no wasted text, making it efficient and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (which likely covers return values), no annotations, and high schema coverage, the description is mostly complete. It explains the tool's purpose, return metadata, optional PDF download, and ID formats. However, it could benefit from more behavioral details like error cases or storage implications, but the output schema reduces the need for return value explanation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description adds value by explaining flexible ID formats (e.g., '2301.12345', 'arXiv:2301.12345', '2301.12345v1'), which provides semantic context beyond the schema's generic description. However, it does not elaborate on the parameters beyond this, so it meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Get' and resource 'detailed information about a specific arXiv paper', specifying it returns complete metadata and optionally downloads the PDF. It distinguishes from sibling tools like 'download_paper' (which likely focuses only on downloading), 'list_downloaded_papers' (which lists already downloaded items), and 'search_papers' (which searches multiple papers).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for retrieving detailed metadata and optionally downloading a PDF for a specific arXiv paper, but does not explicitly state when to use this tool versus alternatives like 'download_paper' or 'search_papers'. It provides context by mentioning flexible ID formats, which helps in usage, but lacks explicit exclusions or named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.