Server Quality Checklist

Tool Scores

• paper_searchB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses available sources and some behavioral constraints (like year format restrictions and source-specific parameters), but doesn't mention rate limits, authentication needs, error handling, or what the return format looks like. The example helps but doesn't fully describe behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized but not optimally structured. It front-loads the purpose but then uses markdown headings that might not render well in all contexts. The example is comprehensive but lengthy. Some information could be more efficiently organized, though all content appears relevant.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (multiple sources with different parameters), no annotations, and no output schema, the description does a decent job but has gaps. It thoroughly documents parameters but doesn't describe the return format, error conditions, or performance characteristics. For a search tool with 8 different sources and complex parameter interactions, more behavioral context would be helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate fully. It provides extensive parameter semantics: lists all available sources, explains query length constraints, max_results range and default, year format with examples, and source-specific parameters (fetch_details for iacr, kwargs for crossref). The detailed example illustrates complex parameter usage, adding significant value beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches academic papers from multiple sources, providing a specific verb ('search') and resource ('academic papers'). It distinguishes itself from siblings like 'paper_download' and 'paper_read' by focusing on search rather than retrieval or reading. However, it doesn't explicitly contrast with siblings in the description text itself.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. There's no mention of sibling tools (paper_download, paper_read) or when search is appropriate versus downloading or reading papers. The example shows usage but doesn't explain context or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• paper_readA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It describes the tool's behavior (reading and extracting text from PDFs) and includes detailed input constraints and examples, but it does not disclose other behavioral traits like error handling, rate limits, authentication needs, or what happens if extraction fails. The description adds value but lacks comprehensive behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately front-loaded with the core purpose, but it includes lengthy examples and formatting that could be condensed. While informative, the structure with bullet points and code blocks is somewhat verbose, reducing efficiency. Every sentence earns its place, but it could be more streamlined.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (multiple sources with different ID formats), the description is quite complete, with detailed examples and input constraints. Since an output schema exists, the description does not need to explain return values. However, it lacks information on error cases or performance limits, leaving minor gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 50% (only 'paper_id' has a description), but the description compensates fully by providing extensive parameter semantics. It explains the meaning of 'searcher' (listing valid sources) and 'paper_id' (with format details and examples for each searcher), adding significant value beyond the input schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'Read and extract text content from academic paper PDFs from multiple sources,' which is a specific verb (read/extract) applied to a specific resource (academic paper PDFs). It distinguishes from sibling tools 'paper_download' and 'paper_search' by focusing on content extraction rather than downloading or searching.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (for reading/extracting text from academic papers) and implies alternatives through sibling tool names, but it does not explicitly state when to choose this tool over 'paper_download' or 'paper_search.' The examples show specific use cases for different sources, which helps guide usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• paper_downloadA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes what the tool does (downloads PDFs), mentions platform-specific constraints (Google Scholar limitation), and specifies the return format ('List of paths to the downloaded PDF files'). It doesn't cover potential errors, rate limits, or authentication needs, but provides substantial operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (Input Constraints, Paper ID formats, Returns, Example) and front-loaded with the core purpose. While comprehensive, some details like the extensive Semantic Scholar formats could be slightly condensed, but overall it's efficient and informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of multiple academic sources with different ID formats, no annotations, and an output schema (which handles return values), the description is complete. It covers purpose, usage constraints, parameter details with examples, return information, and includes a practical example, leaving no significant gaps for the agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must fully compensate. It provides extensive parameter semantics: it explains the 'searcher' parameter with supported platforms and format requirements for 'paper_id' across multiple sources, including detailed examples. This adds crucial meaning beyond what the bare schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Download academic paper PDFs from multiple sources.' This is a specific verb ('download') + resource ('academic paper PDFs') + scope ('from multiple sources'), and it distinguishes from sibling tools like 'paper_read' and 'paper_search' which likely have different functions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool (to download PDFs from academic sources), and it explicitly mentions that Google Scholar doesn't support direct PDF downloads, suggesting an alternative approach. However, it doesn't explicitly compare usage with sibling tools like 'paper_search' or specify when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.