Server Quality Checklist

Tool Scores

• analyze_paper_trendsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the tool analyzes trends but doesn't describe what the analysis entails, how results are returned, or any constraints like processing limits. This is inadequate for a tool with an output schema, as it leaves key behaviors unspecified.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with the main purpose stated first followed by parameter details. Each sentence adds value, such as specifying the source for 'papers' and the analysis options. It avoids redundancy and is efficiently structured, though it could be slightly more polished in formatting.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema, the description doesn't need to explain return values, but it lacks details on behavioral aspects like analysis depth or limitations. With no annotations and low schema coverage, it partially compensates with parameter semantics but falls short in providing a complete operational context, making it minimally adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds significant meaning beyond the input schema, which has 0% description coverage. It explains that 'papers' should be a 'List of papers from search_papers results' and lists the possible values for 'analysis_type' ('authors', 'keywords', 'timeline', 'categories'), clarifying their semantics. This compensates well for the schema's lack of documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool 'Analyze trends in a collection of papers,' which provides a clear verb ('analyze') and resource ('collection of papers'), but it's vague about what 'trends' specifically means. It doesn't differentiate from siblings like 'search_papers' or 'find_related_papers,' which also involve paper analysis, leaving the purpose somewhat ambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description offers no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, such as needing papers from 'search_papers' results, or compare to siblings like 'export_search_results' or 'search_by_author' for different analysis needs. This lack of context makes it hard for an agent to choose appropriately.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• export_search_resultsB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It describes the tool as an exporter with format options and save locations, but lacks details on permissions, rate limits, error handling, or what happens during export (e.g., file creation, overwriting). For a tool with no annotations and potential side effects (file system writes), this is inadequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded with the main purpose. The parameter explanations are concise and directly relevant. However, the colon-prefixed format for parameters is slightly unconventional and could be more integrated, but it doesn't waste space. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (4 parameters, nested objects, no annotations, but has an output schema), the description is moderately complete. It covers input semantics well but lacks behavioral context and usage guidelines. The output schema exists, so return values needn't be explained, but overall completeness is adequate with clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds significant meaning beyond the schema by explaining each parameter's purpose: 'results' as search results, 'format' with specific options, 'filename' as output name, and 'save_path' as directory. This clarifies semantics well, though it doesn't cover all nuances like default behaviors or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Export search results to various formats.' It specifies the verb ('export') and resource ('search results'), but doesn't explicitly differentiate from sibling tools like 'download_paper' or 'analyze_paper_trends', which might have overlapping functionality. The purpose is clear but lacks sibling differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides minimal usage guidance. It mentions that 'results' should come from 'search_papers or other search functions,' which gives some context, but doesn't specify when to use this tool versus alternatives like 'download_paper' or how it relates to other export or analysis tools. No explicit when/when-not scenarios or alternatives are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_related_papersB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions the core functionality but lacks critical details: it doesn't specify whether this is a read-only operation, what data sources are used, potential rate limits, authentication requirements, or how similarity is calculated (e.g., algorithm, keyword extraction method). The description is insufficient for a tool with 4 parameters and no annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured with a clear purpose statement followed by well-organized parameter explanations. Each sentence serves a specific function: the first defines the tool's core functionality, and the subsequent lines document parameters without redundancy. The text is front-loaded with the most important information and wastes no words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there's an output schema (which handles return values), no annotations, and 4 parameters with 0% schema description coverage, the description does an adequate job explaining parameters but falls short on behavioral context. It covers the basic 'what' but lacks the 'how' and operational constraints needed for a tool that performs similarity-based searches. The presence of an output schema prevents this from being a lower score.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description includes parameter documentation with ':param' annotations that explain each parameter's purpose, which adds significant value beyond the input schema (which has 0% description coverage). It clarifies that 'paper_title' is the reference paper, 'max_results' controls output quantity, 'similarity_threshold' defines a minimum score range, and 'category' is an optional filter. However, it doesn't explain what 'similarity score' means in practice or provide examples of valid categories.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Find papers related to a given paper title using keyword similarity.' It specifies the verb ('find'), resource ('papers'), and mechanism ('keyword similarity'), but doesn't explicitly differentiate it from sibling tools like 'search_papers' or 'search_by_author' beyond mentioning the specific input approach.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention sibling tools like 'search_papers' (which might search by broader criteria) or 'get_paper_details' (which might retrieve specific paper metadata), leaving the agent to infer usage context from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_paper_detailsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states what the tool does but doesn't describe any behavioral traits such as whether it's read-only, potential rate limits, authentication needs, error handling, or what 'detailed information' entails beyond the basic operation. This leaves significant gaps in understanding how the tool behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise and well-structured: the first sentence clearly states the purpose, and the second sentence provides essential parameter details with an example. Every sentence earns its place with no wasted words, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there's an output schema (which handles return values), a single parameter with good semantic coverage in the description, and no annotations, the description is minimally adequate. However, it lacks context on behavioral aspects and usage guidelines relative to siblings, leaving room for improvement in completeness for effective tool selection.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful context for the single parameter 'arxiv_id' by explaining it's an ArXiv ID and providing an example format ('e.g., '2301.12345''). Since schema description coverage is 0% and there's only one parameter, this effectively compensates by clarifying the parameter's purpose and format, though it doesn't cover edge cases or validation rules.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get detailed information') and resource ('about a specific paper by ArXiv ID'), making the purpose immediately understandable. However, it doesn't explicitly differentiate from sibling tools like 'search_papers' or 'find_related_papers' that might also retrieve paper information but with different scopes or methods.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With siblings like 'search_papers' and 'find_related_papers' available, there's no indication that this tool is specifically for retrieving details when you already have an ArXiv ID, versus searching or finding related papers through other means.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_by_authorB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. While 'search' implies a read-only operation, the description doesn't address important behavioral aspects: whether authentication is required, rate limits, pagination behavior, what happens when no results are found, or the format/structure of returned results. The description only states what the tool does, not how it behaves.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly structured and concise. The purpose is stated clearly in the first sentence, followed by a well-organized parameter documentation section. Every sentence earns its place - no redundant information, no unnecessary elaboration. The parameter documentation uses a consistent format that's easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there's an output schema (which handles return values), no annotations, and the description provides excellent parameter documentation, the main gap is behavioral context. The description adequately covers what the tool does and how to use its parameters, but doesn't address operational aspects like authentication, error conditions, or performance characteristics that would help an agent use it effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description provides excellent parameter semantics that fully compensate. Each parameter is clearly documented with its purpose and format examples (e.g., 'YYYY-MM-DD or YYYY' for dates, 'cs.SE', 'cs.AI' for categories). The description adds substantial value beyond what the bare schema provides, explaining what each parameter means and how to format values.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search papers by a specific author.' This is a specific verb ('search') with a clear resource ('papers') and target ('by a specific author'). However, it doesn't explicitly differentiate from sibling tools like 'search_papers' - the agent must infer that this tool is author-specific while 'search_papers' might be more general.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With sibling tools like 'search_papers' and 'find_related_papers' available, the agent receives no explicit direction about when author-based searching is preferred over other search methods or when this tool should be avoided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_papersA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adds some context: it mentions natural language query parsing and date/year overriding behavior, which are useful beyond basic search functionality. However, it doesn't cover important aspects like rate limits, authentication needs, error handling, or pagination behavior, leaving gaps for a tool with 6 parameters.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded with the core purpose. The parameter explanations are organized clearly with :param notation, though this format is more technical than natural language. Every sentence adds value, with no wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (6 parameters, no annotations, but with output schema), the description is reasonably complete. It explains all parameters thoroughly and mentions natural language processing capabilities. The output schema existence means return values don't need explanation, but behavioral aspects like rate limits or errors could be more detailed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must fully compensate. It provides detailed semantic explanations for all 6 parameters, including query parsing behavior, date format options, sorting logic, and category examples. This adds significant value beyond the bare schema, making parameter purposes clear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search for papers on arXiv.' It specifies the resource (papers) and the platform (arXiv), but doesn't explicitly differentiate it from sibling tools like 'search_by_author' or 'find_related_papers' in terms of search scope or methodology.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'search_by_author' or 'find_related_papers.' It mentions natural language query parsing, which hints at usage context, but lacks explicit when/when-not instructions or comparisons to sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• download_paperA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively explains key behaviors: the file is saved locally, and in stateless environments, it's temporary and will be deleted on server restart/sleep. This covers important operational constraints, though it doesn't mention error handling, rate limits, or authentication needs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose in the first sentence, followed by a critical behavioral note, then parameter details. Every sentence adds value: the purpose, the temporary file warning, and parameter explanations. No wasted words, and the structure logically progresses from what to why to how.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 2 parameters with 0% schema coverage and no annotations, the description does an excellent job explaining parameters and key behavioral traits (temporary files). The presence of an output schema means return values don't need description. It could slightly improve by mentioning error cases (e.g., invalid IDs) or success confirmation, but it's largely complete for this tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must fully compensate. It adds crucial semantic context for both parameters: 'arxiv_id' is explained with an example format ('e.g., '2301.12345''), and 'directory' specifies the save location and includes a default value mention. This goes well beyond the bare schema, making parameter usage clear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Downloads the PDF of a paper') and resource ('paper'), distinguishing it from sibling tools like 'get_paper_details' (which likely retrieves metadata) or 'search_papers' (which searches). It explicitly mentions the output format (PDF) and destination (local directory), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when a PDF download is needed, but provides no explicit guidance on when to use this tool versus alternatives like 'get_paper_details' (which might return metadata without download) or 'export_search_results' (which could export search data). The note about temporary files in stateless environments offers some context but doesn't address tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.