Server Quality Checklist

Tool Scores

• search_arxivA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It mentions 'Supports Boolean-like searches via keyword' but does not disclose behavior like pagination, error handling, or any side effects. The behavioral transparency is moderate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with three sentences, front-loading the main action. No superfluous text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 7 parameters and an output schema, the description explains the return format (numbered list with fields) but lacks details on parameter usage like 'start' and 'max_results' defaults. It is moderately complete given the output schema exists.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It mentions keyword, author, category, and date range but does not explain parameters like 'start', 'max_results', or the exact format for dates. The description adds some meaning but is incomplete for all 7 parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches arXiv papers by various criteria like keyword, author, category, and date range. It specifies the resource and filtering options. However, it does not differentiate from the sibling tool 'search_papers', which likely has overlapping functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use when the user wants to find papers...' providing clear usage context. It does not mention when not to use or alternatives, but the guidance is sufficient for correct invocation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_papersA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It discloses search scope (all fields) and hints at simplicity relative to search_arxiv, but omits behaviors like pagination, sorting, or error handling. Adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, well-structured with purpose first. Efficient but could be slightly more concise by omitting redundant clarifications like 'just a search query plus...'.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having an output schema, description covers only 3 of 4 parameters, missing max_results entirely. No guidance on category values or date_from format. The tool has moderate complexity (4 params) but description leaves significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet description only adds meaning for query, category, and date_from. It fails to explain max_results parameter or the expected format for date_from (e.g., YYYY-MM-DD). Parameter semantics are incomplete.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Search papers by query text, with optional category filter and date range' and mentions it searches across all fields. It distinguishes itself from search_arxiv, providing a specific verb and resource.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly tells when to use: 'Use when the user provides a natural language query like...' and contrasts with search_arxiv. This gives clear context for selecting this tool over siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_recentA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries the burden. It discloses sorting behavior (by submission date, newest first) and gives example categories. However, it does not mention any limits, pagination, authentication, or rate limits. Adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, concise and front-loaded. Every sentence adds value: first states purpose, second gives usage guidance. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple retrieval tool with 2 params, the description is mostly complete. It covers the main purpose, sorting, and category examples. Minor gap: does not explain the 'max_results' parameter, but the default is visible in schema. Output schema exists, so return details are not needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%. Description provides examples for 'category' (e.g., cs.AI, quant-ph) which adds context. However, 'max_results' parameter is not explained, only a default is shown in schema. Partial improvement via category examples.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves the most recent papers in a given arXiv category, sorted by submission date. It uses a specific verb ('Get') and resource ('recent papers'), and distinguishes from sibling tools by focusing on recency and sorting.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when to use ('when the user wants to see the latest submissions', 'Ideal for 'what is new''). Does not explicitly exclude other use cases or mention alternatives, but the context of sibling tools implies its niche.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_paperA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden. It mentions that IDs can be with or without version suffix, which is helpful. It does not disclose error handling or rate limits, but for a simple read operation these are less critical. It lacks explicit mention of being read-only, but it's implied.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, no wasted words. Front-loaded with action and example. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that an output schema exists (not shown but stated), the description doesn't need to explain return values. It covers tool purpose, usage context, and parameter formatting completely for a simple fetch operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It does so excellently by providing an example ID format (e.g. 2301.07041) and explaining the parameter's purpose (arxiv_id) including version suffix behavior.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it fetches a single paper by arXiv ID and lists the metadata returned (title, authors, etc.). It distinguishes itself from sibling tools like search_arxiv (search) and get_recent (recent papers) by focusing on a specific ID lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says when to use: when the user provides a specific arXiv ID and wants full metadata. It does not explicitly state when not to use, but the context makes it clear that this is for individual paper retrieval, not for search or listing.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fetch_pdfA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries full burden. It discloses table extraction sources and fallback order, and clarifies that the tool extracts full text content.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise with three sentences, each adding essential information. It is front-loaded with the main action, followed by extraction details and usage guidance.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has one required parameter and an output schema likely describes return format, the description covers purpose, usage, parameter example, and extraction details, making it complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has no parameter description, so the description compensates by providing a format example ('e.g. 2301.07041') and stating the ID is required. It adds meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool downloads the PDF and extracts full text with tables formatted as markdown. It distinguishes itself from siblings by specifying it is for reading full paper text, not just the abstract.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says 'Use when the user needs to read the full paper text, not just the abstract,' providing clear context for use. However, it does not directly name sibling tools as alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.