Server Quality Checklist

Tool Scores

• unpaywall_fetch_pdf_textA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions key behaviors like downloading, extracting text, and precedence rules, but lacks details on error handling, rate limits, authentication needs (beyond the email parameter), or what 'best OA PDF' means. This provides basic context but is incomplete for a tool with potential external dependencies.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose ('Download and extract text') and includes essential qualifiers. Every word earns its place, with no wasted text, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description provides basic purpose and parameter context but lacks completeness. It doesn't explain return values (e.g., text format, error responses) or behavioral details like rate limits, which are important for a tool interacting with external services. This is adequate but has clear gaps for effective agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all parameters. The description adds no additional semantic meaning beyond what's in the schema (e.g., it doesn't explain format details or usage nuances). This meets the baseline for high schema coverage but doesn't enhance parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Download and extract text') and the resource ('best OA PDF for a DOI, or from a provided PDF URL'), making the purpose understandable. However, it doesn't explicitly differentiate from sibling tools like 'unpaywall_get_fulltext_links' which might provide links rather than extracted text, so it misses full sibling distinction.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by mentioning alternatives ('DOI, or from a provided PDF URL') and precedence rules ('pdf_url takes precedence over DOI'), but it doesn't explicitly state when to use this tool vs. siblings like 'unpaywall_get_by_doi' or provide clear exclusions. This leaves some ambiguity in tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• unpaywall_search_titlesA
  3.5/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions 'Supports optional is_oa filter and pagination (50 results per page)', which provides useful operational context about filtering capabilities and pagination behavior. However, it doesn't address important behavioral aspects like rate limits, authentication requirements (beyond the optional email parameter), error conditions, or what the search results actually contain.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise (two sentences) and front-loaded with the core purpose. Every sentence earns its place: the first establishes what the tool does, and the second provides key operational context about filters and pagination. There's zero waste or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with 4 parameters, 100% schema coverage, and no output schema, the description provides adequate but minimal context. It covers the basic purpose and mentions key operational features (filtering, pagination), but doesn't explain what the search results contain, how comprehensive the search is, or any limitations. Given the lack of annotations and output schema, more completeness would be helpful for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 100% schema description coverage, the input schema already fully documents all 4 parameters. The description adds minimal value beyond what's in the schema - it mentions the optional is_oa filter and pagination (50 results per page), but these are already clearly documented in the parameter descriptions. The baseline score of 3 is appropriate when the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Search Unpaywall for article titles matching a query') and specifies the resource ('article titles'), making the purpose immediately understandable. However, it doesn't explicitly differentiate this tool from its siblings (unpaywall_fetch_pdf_text, unpaywall_get_by_doi, unpaywall_get_fulltext_links), which appear to have different functions but could potentially overlap in some search contexts.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context through 'Search Unpaywall for article titles matching a query' and mentions optional filters/pagination, but doesn't explicitly state when to use this tool versus alternatives like unpaywall_get_by_doi (for DOI-based lookups) or unpaywall_get_fulltext_links (for link retrieval). No explicit when-not-to-use guidance or prerequisite information is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• unpaywall_get_by_doiA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses key behavioral traits: the tool requires an email address (via env or argument) for identification to Unpaywall, implying authentication needs. However, it lacks details on rate limits, error handling, or response format, leaving gaps in behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded, with two sentences that efficiently convey purpose, input formats, and requirements without any wasted words. Every sentence adds essential information, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is adequate but incomplete. It covers authentication needs and input formats but lacks details on output (metadata structure), error cases, or performance constraints, which would be helpful for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters thoroughly. The description adds minimal value beyond the schema by mentioning the 'doi:' prefix as an accepted format and noting the email requirement, but does not provide additional syntax or usage details for parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Fetch Unpaywall metadata') and resource ('for a DOI'), distinguishing it from sibling tools like fetching PDF text or searching titles. It specifies the exact input formats accepted (DOI, DOI URL, or 'doi:' prefix), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context on when to use this tool (for fetching metadata by DOI) and mentions an alternative (using the 'email' argument vs. env variable), but does not explicitly state when not to use it or differentiate from siblings like 'unpaywall_get_fulltext_links' beyond the metadata focus.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• unpaywall_get_fulltext_linksA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It describes what the tool returns (links and metadata) but does not mention rate limits, authentication needs (though 'email' is optional), error handling, or data freshness. It adds basic context but lacks details on operational traits like performance or constraints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core functionality ('return best open-access links') and includes key details (PDF URL, open URL, metadata) without waste. Every word earns its place, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 parameters, no output schema, no annotations), the description is reasonably complete. It specifies the input (DOI) and output (links and metadata), but without an output schema, it could benefit from more detail on return values (e.g., structure or examples). It covers the essentials but has minor gaps in output clarification.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters ('doi' as required string and 'email' as optional override). The description adds no additional meaning beyond what the schema provides, such as format examples or usage tips for parameters. Baseline 3 is appropriate when schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('return best open-access links') and resource ('given a DOI'), distinguishing it from siblings like 'unpaywall_fetch_pdf_text' (which extracts text) and 'unpaywall_search_titles' (which searches titles). It explicitly mentions the output includes 'best PDF URL and open URL' plus metadata, making the purpose highly specific and differentiated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage context by specifying 'given a DOI' for retrieving open-access links, but it does not explicitly state when to use this tool versus alternatives like 'unpaywall_get_by_doi' (which might return different data) or 'unpaywall_fetch_pdf_text' (which focuses on text extraction). It provides clear context but lacks explicit exclusions or named alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.