Server Quality Checklist

Tool Scores

• get_paper_promptA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden. It only mentions 'precise interpretation of mathematical expressions' as a rationale but omits behavioral details like authentication, rate limits, or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence with front-loaded recommendation. No wasted words, though it could have a slightly more structured format (e.g., separating action from rationale).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given low complexity (1 param, no output schema), the description covers purpose and context. However, it lacks details on return format (e.g., raw LaTeX text), which would improve completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema coverage (arxiv_id described in schema), the description adds no extra meaning beyond the schema. The parameter is self-explanatory, so a baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool fetches the full LaTeX source of an arXiv paper, using specific verb 'fetch' and resource 'full LaTeX source'. It distinguishes from siblings like get_paper_abstract and get_paper_section by focusing on complete source.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'Recommended default' implies a usage context and contrasts with other tools for different needs (abstract, sections). However, it does not explicitly state when not to use it or provide exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_paper_abstractA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It implies a safe, read-only operation without side effects. However, it does not disclose behavior on invalid arxiv_ids or mention any API rate limits or authentication, which would be helpful. Still, the purpose is simple and well-communicated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences: one stating the core function and one specifying usage context. Every word earns its place; no redundancy or fluff. It is compact and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one required parameter, no output schema, no nested objects), the description fully covers what the agent needs: purpose, usage guidelines, and parameter meaning. It is complete for the given complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (the parameter 'arxiv_id' is described with an example). The tool description adds no additional semantics beyond the schema, hitting the baseline. No further parameter details are necessary for this simple tool.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description explicitly states it retrieves 'just the abstract of an arXiv paper', specifying the resource (arXiv paper) and scope (abstract only). It distinguishes from sibling tools like get_paper_section by emphasizing 'just the abstract' and its use case as a quick preview.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear guidance on when to use ('quick preview when the user hasn't read the paper yet') and when not to use ('not when they provide an arXiv ID to discuss a paper'). This contrasts with sibling tools that likely focus on sections or prompts.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_paper_sectionA
  4.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so description carries full burden. It describes a read operation but lacks details on error handling, prerequisites (e.g., paper must be cached), or what happens if the section path is invalid. Basic but not misleading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no extraneous text. Action verb and resource are front-loaded. Every sentence serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with two parameters and no output schema. The description covers purpose, usage guidance, and parameter context adequately. No gaps remain given the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Both parameters are fully documented in the schema (100% coverage). The description adds value by instructing to use list_paper_sections for valid section_paths, going beyond schema details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Get a specific section of an arXiv paper by section path.' It distinguishes from sibling tools like get_paper_abstract and list_paper_sections.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit when-to-use ('when the full paper is too long' or 'user wants to focus') and instructs to use list_paper_sections first, differentiating from alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_paper_sectionsA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so the description carries the burden. It clearly states the tool lists headings, implies read-only behavior, and provides context on its utility. No hidden side effects are mentioned, but for a list operation this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the verb and resource, no wasted words. The purpose and usage are conveyed efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simplicity (one parameter, no output schema), the description fully covers what the tool does and when to use it. No missing information for effective selection and invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Only one parameter with 100% schema coverage. The description adds a concrete example ('2403.12345'), which clarifies the expected format beyond the schema's description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists section headings of an arXiv paper, and distinguishes it from sibling tools like get_paper_section by clarifying it's for identifying which sections to fetch individually.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly explains when to use: when the full paper is too long for context, to identify sections for individual retrieval. This guides the agent on when to prefer this over other tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.