Server Quality Checklist

Tool Scores

• search_papersC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must carry the behavioral disclosure burden, but it only states that the tool searches across multiple sources. It does not reveal rate limits, authentication needs, result aggregation behavior, or whether it is read-only.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no wasted words. It lists specific sources, making it moderately informative while staying concise, though it could have spent more space on parameter semantics.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (7 parameters, no output schema, no annotations), the description is insufficient. It provides no information about return format, query syntax, filtering behavior, or how to effectively use the tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description does not explain any parameters beyond the vague hint of 'multiple sources.' No details are given for sort, limit, year_from, year_to, open_access_only, or how the sources parameter works.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Search') and resource ('scholarly literature') and names five concrete sources (arXiv, OpenAlex, Crossref, Semantic Scholar, PubMed), clearly distinguishing this search tool from sibling tools like get_paper and list_sources.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It does not mention exclusions, scenarios, or prerequisites, leaving the agent to infer appropriate usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_paperA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of behavioral disclosure. It mentions working for paywalled papers (useful) and the return contents, but does not disclose error behavior, rate limits, or authentication needs. It implies a read-only operation but does not state side effects.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single well-structured sentence that is front-loaded with the core purpose and includes a useful qualifier (paywalled). No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  While the description clarifies the main return values and a notable capability, the tool has 4 parameters with no schema descriptions and no output schema. Missing details about id_type, how citations/references are formatted, and error handling make the description incomplete for fully understanding the tool's behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It mentions 'references' and 'citations' but does not map them to the include_references and include_citations parameters, nor does it explain the id_type parameter or identifier formats. The schema has titles and defaults, but the description adds little meaning beyond hinting at the main identifier.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Resolve' and the resource 'one paper by identifier', listing the return contents (metadata, references, citations). This distinguishes it from siblings like search_papers (which discovers papers) and list_sources (which lists sources).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you have a specific paper identifier and need its details and citation network. It highlights that it works for paywalled papers, which is a practical consideration, but it does not explicitly contrast with search_papers or list_sources.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_sourcesA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly mentions that the output includes coverage, key requirements, and rate limits, which gives the agent useful context about limitations and prerequisites. It does not contradict any annotations since none exist.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that states the action and key output details. Every word contributes value, with no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter listing tool with an output schema present, the description is complete. It clearly indicates what the tool returns (sources with coverage, requirements, and rate limits) and differentiates from sibling tools. No additional context is necessary for a task of this simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the schema is trivially complete. Per the rubric, zero params warrant a baseline score of 4. The description adds no parameter-specific meaning, but none is needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('List'), resource ('scholarly sources'), and scope ('the server can search'). It also specifies the included details (coverage, key requirements, rate limits), which distinguishes it from sibling tools like search_papers and get_paper.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool: to discover the scholarly sources available for searching. It implies usage before searching or to understand source capabilities. It doesn't explicitly name alternatives or exclusions, but the listing nature is self-evident and distinguishable from the search and get siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.