Server Quality Checklist

Tool Scores

• download_articleB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It mentions the download action and resolution methods, but lacks details on permissions, rate limits, file handling (e.g., where the PDF is saved), or error conditions beyond a generic 'structured error JSON.' For a tool with no annotations, this is insufficient to guide safe and effective use.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by structured sections for args and returns. It's efficient with no wasted sentences, though the 'Args' and 'Returns' labels are slightly redundant since the schema covers this. Overall, it's well-structured and concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations, 0% schema coverage, but an output schema exists (implied by 'Returns'), the description is moderately complete. It covers the basic purpose and parameters, but lacks behavioral details and doesn't fully leverage the output schema to explain return values. For a download tool with no annotations, it should provide more context on execution and outcomes.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It lists the parameters ('title' and 'arxiv_id') and explains they are used for resolution, but doesn't specify format (e.g., arXiv ID pattern), exclusivity, or how conflicts are handled. This adds some meaning beyond the bare schema, but doesn't fully cover the gaps, resulting in a baseline score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Download') and resource ('article as a PDF file'), specifying the format. It distinguishes from siblings like 'get_article_url' (which likely returns a URL) and 'load_article_to_context' (which might load content without downloading), but could be more explicit about the distinction. The purpose is specific but not fully differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by stating 'Resolve by arXiv ID or title,' suggesting it's for retrieving articles from arXiv. However, it doesn't explicitly say when to use this vs. alternatives like 'search_arxiv' (for finding articles) or 'get_details' (for metadata). No exclusions or clear alternatives are provided, leaving some ambiguity.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• load_article_to_contextA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool supports 'partial extraction' via page/character limits and a 'preview' mode for validation, which adds useful behavioral context beyond the schema. However, it doesn't mention error handling (returns 'structured error JSON'), performance characteristics, rate limits, or authentication needs, leaving gaps for a tool with 7 parameters.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a purpose statement followed by 'Args' and 'Returns' sections, making it easy to parse. It's appropriately sized—every sentence adds value, though the 'Args' section could be more integrated into the flow rather than a separate block. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (7 parameters, no annotations, but with an output schema), the description is mostly complete. It covers parameter semantics thoroughly and mentions the return types ('Article text or structured error JSON'). The output schema likely details the return structure, so the description doesn't need to elaborate further. However, it lacks guidance on tool selection among siblings.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage for 7 parameters, the description compensates fully by explaining each parameter's purpose in the 'Args' section. It clarifies that 'title' and 'arxiv_id' are alternative identifiers, pages are '1-based' and 'inclusive', limits are 'hard caps', and 'preview' validates availability. This adds significant meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Load the article text into context' with support for 'title or arXiv ID resolution and partial extraction'. This specifies the verb (load), resource (article text), and key capabilities. However, it doesn't explicitly differentiate from sibling tools like 'download_article' or 'get_details', which likely have overlapping functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With siblings like 'download_article', 'get_article_url', 'get_details', and 'search_arxiv', the agent has no indication of which tool to choose for loading article text versus downloading files, getting URLs, retrieving metadata, or searching. No prerequisites or exclusions are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_arxivA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that 'only parameters explicitly provided will be included in the search query' and specifies the return format as JSON with titles as keys and arXiv IDs as values. However, it lacks details on rate limits, authentication needs, or error handling beyond a vague mention of 'structured error JSON string.'

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, starting with the core purpose. The Args and Returns sections are structured but slightly verbose; sentences like 'This function allows for flexible querying' could be trimmed without losing value, keeping it efficient overall.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 6 parameters, no annotations, and no output schema, the description is moderately complete. It covers parameter semantics and return format but lacks details on behavioral aspects like rate limits or error structures. For a search tool with multiple parameters, it provides a baseline but misses advanced context needed for optimal agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It effectively explains all 6 parameters: all_fields searches across all metadata, title searches titles, author filters by author, abstract searches abstracts, start is for pagination, and max_results limits results. This adds clear meaning beyond the bare schema, though it could detail format specifics like author name conventions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'performs a search query on the arXiv API' and 'returns matching article metadata,' specifying both the action (search) and resource (arXiv articles). It distinguishes from siblings like download_article and get_details by focusing on search rather than retrieval or downloading.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like get_details or load_article_to_context. It mentions 'flexible querying' but offers no explicit when/when-not instructions or comparisons to sibling tools, leaving usage context unclear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_detailsA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the tool retrieves metadata and returns JSON or error JSON, but does not disclose critical behavioral traits such as whether it requires authentication, rate limits, what happens if both parameters are null, or how it handles ambiguous inputs. For a read operation with zero annotation coverage, this leaves significant gaps in understanding the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with the core purpose stated first, followed by structured sections for arguments and returns. Every sentence adds value, but the 'Args' and 'Returns' sections could be integrated more seamlessly into the flow. It avoids redundancy and is efficient overall.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no annotations, but with an output schema), the description is mostly complete. It explains the purpose, parameters, and return format. The output schema exists, so the description need not detail return values, but it could better address behavioral aspects like error handling or input constraints. It covers the essentials but has minor gaps in context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaning by explaining that 'title' refers to the article title and 'arxiv_id' to the arXiv ID, which clarifies the semantics beyond the schema's type definitions. However, it does not specify format requirements (e.g., arXiv ID pattern), mutual exclusivity, or default behaviors, leaving some ambiguity. With two parameters and low schema coverage, this provides basic but incomplete compensation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Retrieve') and resource ('metadata of an article'), and distinguishes it from siblings by specifying it returns metadata rather than downloading files, getting URLs, loading content, or searching. It explicitly mentions retrieval by title or arXiv ID, making the scope unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (to retrieve article metadata by title or arXiv ID), but does not explicitly state when not to use it or name alternatives among the sibling tools. It implies usage for metadata retrieval rather than other operations, but lacks explicit exclusions or comparisons.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_article_urlA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses the return behavior ('URL that can be used to retrieve the article, or structured error JSON'), which is helpful, but does not mention rate limits, authentication needs, or other operational traits like response formats or error handling details beyond the basic return statement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, followed by structured sections for Args and Returns. Every sentence adds value: the first states the action, the second explains parameters, and the third specifies outputs, with no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no annotations, but with an output schema), the description is mostly complete. It covers purpose, parameters, and return values, but lacks details on behavioral aspects like error conditions or usage prerequisites. The output schema likely handles return structure, so the description's focus on semantics is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaning by explaining that 'title' is the article title and 'arxiv_id' is the arXiv ID with examples (e.g., 1706.03762 or arXiv:1706.03762v7), clarifying usage beyond the schema's basic types. However, it does not detail constraints like format requirements or mutual exclusivity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Retrieve'), resource ('direct PDF URL of an article on arXiv.org'), and scope ('by title or arXiv ID'), distinguishing it from siblings like download_article (which downloads content) or get_details (which provides metadata).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying 'by title or arXiv ID,' but does not explicitly state when to use this tool versus alternatives like search_arxiv (for broader searches) or load_article_to_context (for loading content). It provides clear input options but lacks explicit exclusions or comparisons.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.