Server Quality Checklist

Tool Scores

• find_oa_pdfA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses return values (OA status, best location, list of locations) and the data source, but does not mention read-only behavior, failure handling, or authorization requirements. This is adequate but not thorough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the main purpose, and every word adds value. There is no redundancy or unnecessary detail.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple single-parameter signature and the presence of an output schema, the description provides a complete account of the tool's function and return structure. It lacks explicit usage guidance relative to siblings, but that is not critical for this simple tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides 100% coverage for the 'doi' parameter, including acceptable formats. The description adds no extra parameter-specific meaning beyond the schema, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: find a legal Open Access PDF for a given DOI via Unpaywall. It uses a specific verb and resource, and the distinction from sibling tools like search_papers and download_pdf is clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is used when you have a DOI and need a legal Open Access PDF. It does not explicitly mention alternatives or exclusions, but the context is clear enough for an agent to decide when to invoke it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_papersA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description takes on the burden of explaining behavior. It discloses that the tool 'Returns a compact list' with specific fields (DOI, title, authors, year, venue, citation count, OA status, top concepts), and mentions relevance-based search. This goes beyond a mere definition, though it omits details like rate limits or authentication requirements.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences long, front-loaded with the main purpose, and contains no redundant or filler content. Every sentence provides distinct value: purpose, return contents, and a filtering hint.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (4 parameters) and the presence of an output schema, the description supplies key information about search behavior and return fields. It lacks explicit usage guidance for when to use this tool over siblings, but overall covers the essential context well.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description adds a minor clarification for year_from/year_to ('constrain the publication year'), but this does not significantly exceed the schema's existing descriptions. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with 'Search academic papers by relevance via OpenAlex', which clearly identifies the verb (search), resource (academic papers), and scope (relevance, OpenAlex). This distinguishes it from sibling tools like get_paper (specific paper retrieval), find_oa_pdf, and download_pdf, which have different purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for searching academic papers by relevance, and gives a parameter hint ('Use year_from/year_to to constrain the publication year'), but does not explicitly state when to choose this tool over siblings or provide exclusions. The guidance is implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_paperA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full transparency burden. It discloses non-obvious behaviors: the abstract is 'reconstructed from OpenAlex' and references are 'enriched from CrossRef.' It does not mention error handling or rate limits, but the provenance details add meaningful context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no wasted words. It efficiently conveys the action, scope, and included metadata fields while keeping provenance details concise.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter) and the presence of an output schema, the description covers the essential context: what it fetches and by which identifier. It does not explicitly expand on usage relative to siblings, but the sibling names and the 'single paper by DOI' scope provide sufficient context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema fully documents the doi parameter with examples and accepted formats (bare string, URL, or prefixed). The description adds no extra parameter semantics beyond restating 'by DOI,' and with 100% schema coverage the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool fetches full metadata for a single paper by DOI, listing specific content areas (abstract, authors, concepts, references). It distinguishes itself from sibling tools like search_papers and download_pdf by specifying 'single paper' and 'by DOI'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when you have a DOI and want full metadata, and the sibling tool list makes the alternative contexts obvious. However, it does not explicitly name alternatives or provide when-not-to-use guidance, so it falls short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• research_topicA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It explains the workflow steps (search, then OA status lookup), the data source (Unpaywall), and a key optimization: 'Abstracts are taken from the search response itself — no extra OpenAlex calls.' This adds valuable behavioral context, though it doesn't address error handling or what happens if Unpaywall fails.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core workflow. Every word earns its place, with no redundant phrasing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (2 params, output schema present), the description adequately covers the workflow, the data source, and the intended output. The existence of an output schema removes the need to describe return values. It could mention limitations, but the schema already handles constraints, making this sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description references the 'query' parameter in the first sentence, clarifying its role, but does not add details beyond the schema's own parameter descriptions. No additional semantics are needed.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states its purpose as a 'Convenience workflow' that searches OpenAlex, enriches results with OA status from Unpaywall, and returns a compact overview. This is a specific verb+resource combination that differentiates it from siblings like search_papers (search only) and find_oa_pdf (find PDFs).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context, calling it a 'Convenience workflow' 'suitable for quickly assessing a research area.' This implies when to use it but does not explicitly name alternatives or exclusions. Since sibling tools are listed, the context is clear enough for an agent to infer the intended use case.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• download_pdfA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden and does so excellently. It discloses SSRF safety validation, PDF magic-marker verification, a 100 MB size cap, and atomic writes that prevent partial files. This goes well beyond typical descriptions and addresses security and reliability concerns.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary purpose, and then efficiently packs usage guidance and safety details. Every sentence earns its place, with no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, prerequisite, safety, size limits, and failure behavior. The existence of an output schema means return values need no description. A slight gap is that it doesn't mention error handling for non-PDF content beyond the magic-marker check, but the given details are otherwise comprehensive for this tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, and the schema already explains each parameter (url, doi, filename) with useful context. The description adds no new parameter-level semantics; it only reinforces that the URL should come from find_oa_pdf. Per calibration, a baseline of 3 is appropriate when the schema covers parameters well.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Download a PDF to the local PDF directory.' This is a specific verb+resource+location combination. It also implicitly differentiates from siblings like find_oa_pdf by focusing on the download step, and explicitly references the typical prerequisite workflow.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit workflow guidance by instructing 'Use find_oa_pdf first to obtain a URL.' This tells the agent when to use this tool in sequence with a sibling. It lacks an explicit 'when not to use' clause, but the context is clear enough for a download tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.