Server Quality Checklist

Tool Scores

• search_arxivB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries the burden. It mentions that keyword supports Boolean-like searches and that results are a numbered list with specific fields. However, it does not disclose behavior like pagination, error handling, or how empty results are handled.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Description is three sentences, front-loaded with purpose. Efficiently covers main parameters and return value. Could be slightly more structured but is generally concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 7 parameters, 0 required, no schema descriptions, and no annotations, the description covers the core search functionality and return fields. It lacks details on parameter format and pagination. The presence of an output schema reduces burden but does not fully compensate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so description must compensate. It explains the purpose of keyword, author, category, and date range parameters, but does not clarify format for dates or categories, and omits max_results and start. The Boolean-like search hint adds some value.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it searches arXiv papers by keyword, author, category, and/or date range. It specifies what is returned (title, ID, authors, date, PDF link). It implicitly differentiates from siblings like fetch_pdf and get_paper by focusing on search, but does not distinguish from search_papers.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Description provides explicit usage context: 'Use when the user wants to find papers matching specific terms...' but does not specify when not to use or mention alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_papersA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description bears full burden. States query searches all fields (title, abstract, authors). However, lacks details on result ordering, pagination, rate limits, or side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four sentences, each with a clear purpose. Could front-load usage guidance slightly, but overall efficient with no extraneous content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Output schema exists, reducing need to describe return values. With 4 parameters and no annotations, the description covers basic usage and search scope but omits important details like default ordering, result limits, and error handling.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but description only elaborates on query ('searches across all fields'). Omits max_results entirely. Category and date_from are mentioned as optional but without additional meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states it searches papers by query text with optional filters. Specifically distinguishes from sibling search_arxiv by calling it 'simpler' and listing the parameters.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises use 'when the user provides a natural language query' and optionally a category/date. Mentions alternative search_arxiv but does not specify when to choose one over the other.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_recentA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It mentions sorting (newest first) and category scope, but does not address rate limits, pagination, error handling for invalid categories, or the default max_results value. The presence of an output schema partially compensates, but more detail would be helpful.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of two concise sentences, with the purpose front-loaded in the first sentence and usage guidance in the second. No unnecessary words or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (2 params, 1 required) and the existence of an output schema, the description is mostly complete. It covers the core functionality and usage context. However, the lack of detail about max_results is a minor omission, and the description could briefly mention that results are limited and search behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 0% description coverage. The description adds meaning for the category parameter with examples (cs.AI, quant-ph), which is helpful. However, it does not mention the max_results parameter at all, leaving its purpose and default behavior undocumented. Since there are only two parameters, missing one is a significant gap.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves the most recent papers in a given category, sorted by submission date, and provides examples. It implicitly distinguishes from sibling tools (e.g., search_arxiv) by focusing on 'latest submissions' and 'what is new' queries, but could be more explicit about differentiation.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use when the user wants to see the latest submissions in a specific arXiv category' and 'Ideal for what is new style queries,' providing clear context for when to use the tool. However, it does not specify when not to use it or list alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_paperA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavior. It mentions fetching and returning metadata, implying a read-only operation. However, it does not explicitly state there are no side effects, rate limits, or error conditions; this gap prevents a higher score.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences that efficiently convey the tool's purpose, usage, and parameter format. No redundant words, well front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists and the tool has only one parameter, the description fully covers what the tool does and returns. The user understands exactly what to expect.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must add meaning. It explains the parameter format (e.g., '2301.07041'), mentions version suffix, and gives an example. This goes beyond the schema's type 'string' alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool fetches a single paper by arXiv ID and lists specific metadata returned (title, authors, abstract, PDF link). It distinguishes from siblings like search_arxiv (search) and fetch_pdf (PDF download) by specifying a single paper lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use: when user provides a specific arXiv ID and wants full metadata. It also mentions ID format and version suffix. Could be improved by stating when not to use (e.g., for search), but the sibling list provides context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fetch_pdfA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations, so description must cover behavior. It discloses extraction sources (ar5iv, LaTeX, PDF/Camelot) and output format. Could mention failure modes but overall good.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each purposeful: purpose, extraction detail, usage guideline with example. No redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers input and output well given one parameter and presence of output schema. Could add error handling but not critical.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but description provides an example format (2301.07041) and states it's required, adding value beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states the tool downloads a PDF and extracts full text with tables in markdown. Distinguishes from siblings by specifying it's for full paper text vs. abstract.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says to use when user needs full paper text, not just abstract. Implies alternative (get_paper) but does not name it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.