Server Quality Checklist

Tool Scores

• get_paper_by_idB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It only says 'retrieves full details' without specifying what 'full details' means, how errors are handled, or any permissions/rate limits. This is a significant gap for a tool with zero annotation coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, clear sentence with no wasteful words. It is appropriately brief for a simple tool, though it could be slightly expanded with useful details while maintaining conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter retrieval tool with no output schema and no annotations, the description is minimally adequate but leaves major gaps: it doesn't describe the return payload structure, behavior when the paper is not found, or any limitations. The phrase 'full details' is vague and incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has a single parameter 'arxiv_id' with no description, and the description adds only 'using its arXiv ID', which essentially repeats the property name. It provides no format examples, constraints, or clarification on how the ID should be formatted (e.g., with or without the 'arXiv:' prefix).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the verb 'retrieves', the resource 'full details for a single paper', and the input method 'using its arXiv ID'. This differentiates it from sibling tools like get_papers_by_ids which handles multiple papers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies that this is the tool to use when you have a specific arXiv ID and want that paper's details, but it does not explicitly state when to use it over alternatives like search_papers or find_papers_by_author. No direct exclusions or alternative references are given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_papers_by_idsA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full responsibility. It indicates a read operation ('Retrieves'), but provides no details about potential edge cases (e.g., invalid IDs, missing papers), rate limits, data source specifics ('from sources like arXiv' is vague), or the exact structure of the returned data.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exactly two sentences: the first sentence concisely states the primary purpose, and the second sentence provides a clear parameter definition with examples. No redundant or unnecessary text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (one parameter), but there is no output schema, and the description does not specify what 'full details' includes (e.g., fields, format) or how errors are handled. This is a gap for a tool that lacks output documentation, though the simplicity mitigates the impact.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only defines an array of strings without semantic meaning. The description compensates by explicitly identifying the parameter as arXiv IDs and giving concrete examples, making the parameter's purpose and format clear beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the verb 'Retrieves', the resource 'full details for a list of papers', and the method 'using their arXiv IDs'. It clearly distinguishes from the singular sibling 'get_paper_by_id' by specifying 'a list' and 'IDs'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is for fetching papers when a list of arXiv IDs is already known, but it does not explicitly compare with sibling tools like search_papers or get_paper_by_id, nor mention when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_papers_by_authorA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It mentions 'recent' and 'sources like arXiv,' but fails to disclose search match behavior, rate limits, error handling, or return format. This is minimal behavioral disclosure for a search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with a concise, actionable sentence followed by a tight args list. Every word earns its place; no extraneous content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (2 params, no output schema), but the description omits return value details and usage guidance relative to sibling tools. Given no annotations and no output schema, the lack of behavioral and return information makes it slightly incomplete, though usable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. The 'Args' section fully explains both parameters: author_name with an example ('Geoffrey Hinton') and max_results with its default. This adds clear meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Finds recent academic papers by a specific author's name from sources like arXiv,' using a specific verb and resource. It distinctively positions this tool as author-based lookup, differentiating it from siblings like search_papers and get_paper_by_id.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for author-based paper searches but does not explicitly contrast with sibling tools such as search_papers or get_latest_from_category. It lacks when-not-to-use guidance, so it only provides implied usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_latest_from_categoryA
  3.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It only states the basic action and parameters, without disclosing return format, rate limits, whether it's read-only, or any other behavioral traits. This is a notable gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with the main purpose, followed by an efficient parameter list. Every sentence adds value with no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with only two parameters, and the description covers the args well. However, there is no output schema and the description does not mention what the return value looks like, leaving some gaps for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description fully explains both parameters with useful details: 'category' includes an example code, and 'max_results' is defined as the maximum number of papers. Since the schema has no descriptions, this fully compensates for the 0% coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function with a specific verb and resource: 'Gets the most recently submitted papers from a specific arXiv category.' It distinguishes from siblings by focusing on category-based retrieval of recent papers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you need the latest papers from a specific arXiv category, but it does not explicitly compare with sibling tools or provide exclusion criteria. It is clear enough but lacks explicit alternative guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_papersA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the arXiv-specific query syntax with an example, explains sort fields and directions, and notes the source. This gives the agent useful behavioral context for constructing searches, though it doesn't describe return format or any edge cases like no results. It is substantial but not exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: a single-sentence overview followed by a clear bulleted list of parameters. Every sentence earns its place, with no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity and the lack of annotations/output schema, the description provides essential context: the source (arXiv), query syntax, and parameter semantics. It doesn't explain the return shape, but the function name and sibling context make 'search' behavior predictable. Overall, it is nearly complete for a search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides only types and defaults, leaving the meaning of parameters entirely to the description. The description explains the query parameter with detailed arXiv syntax and an example, and defines max_results, sort_by (with valid values), and sort_order (with valid directions). This fully compensates for the 0% schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool is a 'General purpose search for academic papers from sources like arXiv' and notes it's 'Best for complex queries.' This uses a specific verb ('search') and resource ('academic papers'), and the 'general purpose' contrast distinguishes it from specialized siblings like find_papers_by_author or get_latest_from_category.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Best for complex queries' provides clear context for when to use this tool, implying it's preferred over the more specialized sibling tools for complex or broad searches. However, it stops short of explicitly naming alternatives or stating when not to use it, so it doesn't fully meet the bar for explicit exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.