Server Quality Checklist

Tool Scores

• unpaywall_export_bibtexA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description carries the burden. It discloses that it uses the 'most recent' results, implying statefulness, but does not mention side effects, prerequisites, or error conditions. Adequate but minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with action and format, no wasted words. Perfectly concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters and no annotations, the description is minimal. It omits the dependency on prior tool calls and the fact that only the most recent results are exported. Output schema exists, so return values are covered, but the overall context of usage is incomplete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters, so baseline is 4. The description adds meaning by specifying what is exported (results of lookup/search) and the format (BibTeX), which is beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool exports the most recent results from unpaywall_lookup or unpaywall_search in BibTeX format. It distinguishes from export_ris by specifying the format, but could be more explicit about the exact resource.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit when-to-use or when-not-to-use guidance. It implies it must follow a lookup or search, but doesn't warn that previous results are overwritten, nor does it compare with export_ris.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• unpaywall_lookupA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided, so description carries full burden. It lists return data but does not disclose potential side effects, authentication needs, rate limits, or error behavior. However, the tool name and context suggest a read-only operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences plus an args line, front-loaded with purpose, then returning details. No redundant text, every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter lookup tool, the description covers purpose, parameter, and output. The output schema further clarifies return values. Missing error handling or edge cases, but overall complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema coverage, the description is essential. It explains the 'doi' parameter with an example and states its purpose ('The DOI to look up'). This adds significant value beyond the schema's 'Doi' title.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states the tool's purpose: 'Look up open-access availability for a scholarly article by DOI.' It specifies the verb and resource, and distinguishes from siblings (export and search tools) by focusing on a specific DOI lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when a DOI is known, but does not explicitly state when to use this tool versus alternatives like 'unpaywall_search'. No guidance on when not to use or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• unpaywall_export_risA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the full burden. It discloses that it exports the most recent results and produces a .ris file, but it does not state behavior when no previous results exist, file naming conventions, or error handling. This is adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no wasted words. The first sentence states the core function, and the second gives a concrete usage instruction. Perfectly front-loaded and efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has no parameters but has an output schema (not shown). The description explains the output file format and a use case, but it does not cover edge cases (e.g., no prior results, multiple results). Given the tool's simplicity, the description is mostly sufficient but misses some context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has zero parameters, and schema coverage is 100%. With no parameters to describe, the description adds no parameter information beyond the schema, but baseline for 0 params is 4. The description naturally cannot add value here.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool exports the most recent unpaywall lookup or search results to RIS format, which is a specific verb and resource. It distinguishes from sibling tools (unpaywall_export_bibtex exports to BibTeX, unpaywall_lookup and unpaywall_search have different functions).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit context: it exports the most recent results and gives instructions to import into Zotero via File -> Import. It implies a prerequisite (prior lookup/search) but does not explicitly state when not to use it or alternatives (e.g., use unpaywall_export_bibtex for BibTeX format).

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• unpaywall_searchA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description bears the full responsibility. It discloses result limits (50 per page), pagination, and filtering options, but does not mention authentication, rate limits, or output format. With output schema present, return format is covered, but other behavioral traits are missing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and front-loaded with the core purpose. It uses a two-paragraph structure: a brief statement of function and query syntax, followed by an organized Args list. No superfluous content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the existence of an output schema, the description adequately covers search behavior, query syntax, pagination, and filtering. It does not mention rate limits or API key requirements, which could be useful. Slightly incomplete but still high.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It clearly explains each parameter's purpose, type, and default values with examples (e.g., query: 'e.g. "machine learning"', is_oa: 'true = OA only, false = closed only, omit for all', page: '50 results per page, default 1'). This fully compensates for the missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Search Unpaywall for scholarly articles by title.' This is a specific verb+resource combination, and it distinguishes from siblings (export and lookup tools) by focusing on search functionality.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains the query syntax and pagination, giving clear usage context. While it doesn't explicitly compare with siblings, the purpose and features suggest when to use this tool over alternatives. Slight room for more explicit guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.