Server Quality Checklist

Tool Scores

• download_and_parse_paperA
  4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and does this well. It discloses important limitations: no Markdown files, does not bypass paywalls, publisher HTML is skipped, and only legal OA sources are used. It also lists possible status values, giving agents a clear picture of expected outcomes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, front-loaded with the core purpose, and every sentence adds value. The separation into purpose, limitations, accepted inputs, and status values makes it easy to scan and understand quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is largely complete for a tool with an output schema and full schema-level parameter documentation. It covers success/failure statuses and critical boundary behaviors. It lacks explicit guidance on what the parse output contains or when to prefer sibling tools, but the output schema and sibling names partially compensate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already covers 100% of the parameters with detailed descriptions, so the baseline is 3. The description restates DOI/arXiv acceptance but adds little beyond the schema; it does not meaningfully enrich the semantics of target_dir beyond what the schema already specifies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool downloads an open-access PDF and that it does not create Markdown files, which distinguishes it from a generic downloader. It also identifies accepted input types (DOI or arXiv ID). However, the 'parse' aspect from the name is not directly explained, and sibling differentiation is only implicit.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context: it handles DOI/arXiv identifiers, resolves legal open-access copies, and skips publisher HTML. It does not explicitly say when to use this tool versus extract_sections, search_papers, or get_bibtex, nor does it name alternatives. Usage is implied rather than explicitly guided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_bibtexA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly states that the tool appends to a local file, skips duplicates (status=duplicate), leaves the file unchanged on failure, and enumerates possible statuses (added, duplicate, not_found, error, invalid_path). This is strong transparency for a write action. However, it does not mention whether the target file is created if missing, which is a minor behavioral gap. Hence a 4.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise and well-structured: the first sentence states the core purpose, the second explains duplicate handling and failure behavior, and the third lists status values. Every sentence contributes essential information without fluff. The most important information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the essential functionality and edge cases: fetching, appending, duplicate detection, failure behavior, and status outcomes. Since the tool has an output schema (confirmed by context signals), the status values are likely defined there as well. The only notable omission is the behavior when the target file does not exist (whether it is created), but for a simple two-parameter tool, the description is quite complete. Thus a 4.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%: both parameters (doi and bib_path) have clear descriptions in the schema (e.g., 'DOI string or https://doi.org/... URL' and 'Workspace-relative bibliography path (default references.bib)'). The description adds minimal extra parameter-specific meaning, only reinforcing that the path is local and that the default is used. Since the schema already fully documents the parameters, the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the purpose with a specific verb: 'Fetch the official Crossref/DOI BibTeX entry and append it locally.' It names the resource (Crossref/DOI BibTeX) and the action (append locally), making it distinct from sibling tools like extract_sections or search_papers, which handle unrelated tasks. The purpose is unambiguous and immediately actionable.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly defines what the tool does (fetch and append BibTeX) and its context (workspace-relative path, default references.bib). It does not explicitly state when to use it versus alternatives, but since the sibling tools are clearly for different functions (section extraction, paper search, download/parse), the use case is self-evident. There is no explicit 'when not to use' or alternative comparison, but the absence of overlapping functionality makes a 4 appropriate.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• extract_sectionsA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It discloses key behavioral traits: heading matching is case-insensitive, aliases exist (e.g., Methodology ≡ Methods), missing sections are listed in `unmatched`, status values are enumerated, and it never fetches URLs. It does not, however, describe the output schema in detail, but later output schema may exist. The behavioral notes are substantial and go beyond basic trust.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four compact paragraphs, each earning its place. First line states purpose, second explains the heuristic nature, third lists status values, fourth states a critical constraint. No fluff, front-loaded with the core action, and structured for quick skimming.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has an output schema (implied), so return format is covered. Key operational details are present: matching behavior, status values, and the URL restriction. A slight gap: it doesn't explicitly mention the output structure (e.g., section content vs. just match status), but since output schema exists and the description is otherwise thorough, this is minor. Given the complexity, covers nearly all needed information.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents both parameters. The description adds useful alias semantics: it explains that headings are matched with aliases and gives examples, which enriches `sections`. It also clarifies that `pdf_path` is workspace-relative and must already exist. This goes beyond bare schema, adding practical usage context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States the action ('Return named sections'), the resource ('local PDF'), and the method ('heading heuristics'). Clearly distinguishes from siblings like search_papers (searching) and download_and_parse_paper (downloading + parsing) by scoping to extracting sections from an existing local PDF. The description leaves no ambiguity about its core function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says what it does not do: 'Does not fetch URLs — `pdf_path` must already exist in the workspace.' This prevents misuse when the agent might think it can download. It also clarifies that missing sections do not fail the tool, guiding the agent on expected behavior. Alternatives are not named, but the functional boundary is clearly drawn.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_papersA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden, and it delivers: it discloses that PDFs are not downloaded, that identifier lookups override keyword queries, that arXiv keyword hits are skipped in cited_by mode, that unknown sort values fall back to relevance, and that error_code may be source_unavailable with possibly partial results. This is rich behavioral context beyond the bare operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded, with the core purpose in the first sentence and usage guidance immediately after. Every paragraph adds useful information, though the ranking and identifier-lookup sections are somewhat dense and could be tightened without losing value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the essential context for a 7-parameter, zero-required tool with an output schema: when to use it, ranking semantics, identifier behavior, and possible error codes. It also notes that an empty papers list is still status=ok, which prevents misinterpreting valid empty results. The presence of an output schema means return-value documentation is not the description's responsibility.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds meaning well beyond the schema. For sort, it explains aliases and fallback ('unknown values fall back to relevance'). For query, doi, and arxiv_id, it explains lookup behavior and precedence ('Identifier filters win over a keyword query'). This significantly aids correct invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Search open catalogs (OpenAlex + arXiv) for academic papers.' It also distinguishes this tool from its siblings by stating it 'Does not download PDFs' and 'Use this first when the user needs papers, citations, or a reading list,' making it clear that search is the entry point while download/extraction/bibliography tools are separate follow-ups.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance: 'Use this first when the user needs papers, citations, or a reading list.' It also provides a clear when-not ('Does not download PDFs') and explains how to choose between sort modes ('pass sort=cited_by when the user wants the most cited / famous / popular papers'). It falls short of naming the alternative sibling tools explicitly, but the pipeline context is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.