Server Quality Checklist

Tool Scores

• find_oa_pdfA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description must disclose behaviors. It states the output includes OA status, best OA location, and all OA locations, but lacks details on error handling, network requirements, or rate limits. Some transparency but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences front-load the main action and output details with no redundant information. Every sentence adds value, and the structure is efficient for an AI agent to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one required parameter, no enums, output schema present), the description covers key outputs. However, it omits potential edge cases (e.g., DOI not found) and prerequisites (e.g., internet access), leaving minor gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter 'doi' is already well-described in the input schema (100% coverage). The description adds no additional parameter semantics beyond what the schema provides, so baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool finds a legal Open Access PDF for a DOI via Unpaywall, specifying the verb (find), resource (DOI), and distinct purpose from sibling tools like search_papers or download_pdf.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for locating legally accessible PDFs but does not explicitly guide when to use this tool over alternatives like download_pdf or get_paper. No when-not-to-use or prerequisites mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_papersA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must convey behavioral traits. It states that results are returned as a 'compact list' with specific fields (DOI, title, authors, year, venue, citation count, OA status, top concepts). This gives a clear picture of output. However, it does not disclose the ordering ('by relevance' is vague), pagination limits, data freshness, or whether it requires authentication. The description is adequate but not fully transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences: the first defines purpose and output, the second gives a usage hint. Every word is valuable. No redundancy or filler. The critical information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists (so return values are documented), the description adds enough context: it lists the return fields briefly. It covers the main parameters. It could mention that results are sorted by relevance or that the search uses OpenAlex API, but it's nearly complete for a simple search tool. A slight gap: no mention of how to interpret the fields (e.g., 'top concepts' format).

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% (all four parameters have descriptions in the schema). The description adds only one extra hint: 'Use `year_from`/`year_to` to constrain the publication year.' Since the schema already explains each parameter, the description provides marginal additional meaning. The baseline for high coverage is 3, and the description meets that.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States exactly what the tool does: 'Search academic papers by relevance via OpenAlex.' The verb 'Search' and resource 'academic papers' are specific. The mention of OpenAlex distinguishes it from potential other paper search tools. The list of return fields further clarifies scope. Among siblings, it clearly differs from get_paper (single paper retrieval) and find_oa_pdf (PDF link).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides one specific usage hint: 'Use `year_from`/`year_to` to constrain the publication year.' This is helpful but doesn't offer guidance on when to choose this tool over its siblings (e.g., when to search vs. get a specific paper). No explicit when-not-to-use or alternatives are mentioned, so the guidance is minimal.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_paperA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must disclose behavioral traits. It describes output content (abstract reconstructed from OpenAlex, references from CrossRef) but omits side effects, error handling, authentication needs, or rate limits. Adds value on response content but lacks broader behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that effectively front-loads the action and resource. It lists included items without excess words, though splitting into two sentences could improve readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has one parameter and an output schema (not shown), the description explains what the response includes (abstract, authors, concepts, references). It does not address error conditions or preconditions, but for a fetch-by-key tool, this is mostly complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers 100% of parameters and already describes the DOI format acceptance. The tool description merely mentions 'by DOI' without adding new meaning beyond the schema. Baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Fetch full metadata for a single paper by DOI'. It lists specific components included (abstract, authors, concepts, references), distinguishing it from siblings like search_papers or download_pdf.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when a specific DOI is known and full metadata is needed, but does not explicitly compare with sibling tools or state when not to use it. Usage context is clear but lacks exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• research_topicA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description transparently reveals it searches OpenAlex, uses Unpaywall for OA status, and extracts abstracts from search responses. It does not mention read-only nature, error handling, or rate limits, but the workflow is clearly non-destructive and well-outlined.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core workflow, and every sentence adds value. No unnecessary words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the main workflow, data sources, and abstraction sourcing. With an output schema present, return values are documented elsewhere. Missing details about potential errors or authentication are minor given the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so parameters are well-documented in the schema. The description adds context by linking query to OpenAlex search and mentioning top results, but does not provide deeper semantics beyond what the schema already offers.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool is a convenience workflow that searches OpenAlex for a query, fetches OA status for top results, and returns a compact overview. It distinguishes from sibling tools like search_papers and get_paper by combining search and OA enrichment in a single step.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for quickly assessing a research area but does not explicitly state when to use it versus alternatives like search_papers or get_paper. No exclusions or conditional guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• download_pdfA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description bears the full burden of disclosure. It comprehensively details behavioral traits: SSRF safety validation (refusing private/loopback/metadata IPs), magic marker verification (%PDF-), size cap (PDF_MAX_BYTES, default 100 MB), and atomic write semantics. This fully informs the agent of safety and correctness guarantees.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is four sentences long, front-loaded with the core purpose, and contains zero superfluous words. Every sentence adds critical information about workflow, validation, or guarantees. It is optimally sized for quick consumption.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (reducing need to describe return values) and relatively low complexity (3 parameters, 1 required), the description covers all essential aspects: prerequisite step, safety checks, size limit, and atomicity. It leaves no major behavioral gaps for a download tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description does not add new parameter-level meaning beyond the schema, focusing instead on URL processing behavior. It does not elaborate on how `doi` or `filename` are used beyond the schema descriptions, so no uplift is warranted.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Download a PDF to the local PDF directory'), identifies the resource type (PDF), and distinguishes from siblings by explicitly referencing the prerequisite tool `find_oa_pdf`. The verb-resource combination is specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit when-to-use guidance by instructing practitioners to first use `find_oa_pdf` to obtain a URL. This frames the tool as the second step in a two-step workflow, making its context clear. No alternatives are needed as the sibling tools cover different use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.