Server Quality Checklist

Tool Scores

• search_papersB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description only describes parameters, not behavioral traits like rate limits, authentication, or side effects. It implies a read-only search but does not state it.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Concise block of text with a summary line and parameter list. Could be more structured (e.g., bullet points) but no redundant information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers parameter semantics well, but lacks usage guidelines, output format description, and behavioral context. Adequate but with clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description compensates well by explaining each parameter's purpose and format, e.g., year range examples. However, default values for some parameters like 'year' are not explicitly stated.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description states 'Relevance search for papers on Semantic Scholar', which is a clear verb+resource. However, it does not differentiate from sibling tools like 'search_by_topic'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. Sibling tools like 'fetch_paper_details' and 'search_by_topic' are not referenced.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fetch_paper_detailsA
  3.7/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description should disclose behavioral traits. It only states it 'gets details' but omits safety info (e.g., read-only assumption), error handling, or any limitations. The description is minimal for a tool with no annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise: a single sentence plus an arg clarification. Every word adds value; no wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one param and no output schema, the description adequately covers the input but lacks any mention of return format or additional behavior. It meets minimum viability but has clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds valuable context to the single parameter paper_id, specifying it can be a Semantic Scholar paper ID, DOI, arXiv ID, etc., which is not present in the input schema (0% schema coverage). This significantly aids correct invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Get detailed information about a single paper' with a specific verb and resource, and implicitly distinguishes from sibling tools like get_papers_batch (multiple papers) and search procedures.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for fetching details of one paper but provides no explicit guidance on when to use this tool versus alternatives like get_papers_batch, search_papers, or recommend_papers. No when-not-to-use information.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_papers_batchA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided. Description mentions rate limit efficiency but does not disclose side effects or return behavior. Adequate for a read-only batch fetch, but could be more explicit about non-destructive nature.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three well-structured sentences plus an Args line. Front-loaded with purpose, efficient in word count, no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given one parameter and no output schema, the description covers the main usage, benefit, and parameter format. Could mention maximum batch size in the description, but it's already stated. Appropriate for its simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% coverage, but description adds that paper_ids can be Semantic Scholar paper IDs, DOIs, arXiv IDs, etc. This significantly helps the agent understand acceptable inputs beyond the schema's plain array of strings.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it fetches details for many papers in one request, with a specific batch size limit (up to 500). It distinguishes from the sibling fetch_paper_details by emphasizing the batch aspect.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states it is more efficient than calling fetch_paper_details in a loop, especially under rate limits. Does not specify when not to use it, but context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_by_topicA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that it maps year_start/year_end to a year-range string and mentions defaults and a cap for the limit parameter. However, it does not describe the return format, pagination, or whether the operation is read-only. The disclosure is adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured. The first sentence states the purpose, followed by a clarifying line about the wrapper, and then a clean Args list. Every sentence adds value without verbosity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (6 parameters, 1 required, no output schema), the description covers parameters thoroughly and references search_papers for filter details. It lacks an explanation of the return format, but for a search tool this is acceptable. The description is nearly complete for its purpose.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description compensates well. It explains every parameter: topic as the search query, year_start/year_end as inclusive range, limit with default and cap, fields_of_study as a list of fields, and min_citation_count as a threshold. This adds significant meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches papers by topic with optional year range and filters. It explicitly distinguishes itself as a thin wrapper over search_papers, specifying the mapping of year_start/year_end to the API's year-range string. The verb 'search' and resource 'papers' are clearly identified.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: it is a thin wrapper over search_papers and directs users to search_papers for filter semantics. This implies when to use this tool (when searching by topic with year range) versus search_papers (when needing more complex filters). However, it does not explicitly state when not to use it or list alternative sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• recommend_papersA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It explains it uses citation graph, specifies constraints like default limit and cap at 100. However, it does not explicitly state it is a read-only operation, though it is implied. Minor gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Highly concise: three sentences total (purpose, usage, args), no unnecessary words. Front-loaded with key information. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 3-parameter tool with no output schema, it covers purpose, usage, and all parameters well. Missing description of return format (e.g., list of paper objects) but overall adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description's Args section thoroughly explains each parameter: positive_paper_ids (required), negative_paper_ids (optional), and limit (default 10, capped at 100). This adds significant meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it recommends papers similar to seed papers via the citation graph. It distinguishes itself from sibling tools by contrasting with keyword search, making its unique purpose evident.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly advises using it as a second discovery channel after keyword search, and explains how to seed it with relevant paper IDs from search results. This provides clear when-to-use and how-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.