Server Quality Checklist

Tool Scores

• search_papersA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses several behaviors: deduplication, ranking, no full-text fetch, and graceful handling of individual source failures via warnings. This is substantial but not exhaustive (e.g., auth needs, rate limits).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact, with key search behavior front-loaded and examples in parentheses. It is slightly unstructured but efficient and earns its length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For an 8-parameter search tool with no annotations, the description covers core behavior (source failure, dedupe, ranking) but lacks detailed parameter guidance and does not mention pagination or output shape (though output schema exists). Adequate but with clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema has 0% description coverage. The description only explains 'categories' (arXiv-only filter with examples) and 'languages' (ja/en), but omits semantics for query, limit, offset, sources, year_from, and year_to. Many parameter meanings are left to inference from names.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches Japanese/English academic papers across arXiv and CiNii, returning deduplicated/ranked candidates. This distinguishes it from sibling tools like inspect_papers or collect_papers, which imply different operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides context that search depends on languages and notes it does not fetch full text, implying it is not for full-text retrieval. However, it never explicitly names alternative tools or provides when/when-not criteria relative to siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_collected_papersA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of behavioral disclosure. It usefully discloses that omitting paper_ids returns a list and that the output includes path and SHA-256, but it does not explicitly state read-only safety, error conditions, or required permissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short sentences, front-loaded with the primary purpose, followed by an important conditional usage. No redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with one optional parameter and an output schema is indicated, so the description need not enumerate return values. It covers the main behavior and the conditional list mode; only minor gaps such as error handling and permission requirements remain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains that paper_ids is optional and that omitting it returns a list, adding meaning beyond the schema. However, it does not describe the expected string format of IDs or what happens if IDs are provided but not found in the collection.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool returns local information (path, SHA-256) of collected papers, with a specific verb (返す) and resource (収集済み論文のローカル情報). It also distinguishes itself from siblings by focusing on already-collected papers' local metadata and adding the list behavior when paper_ids is omitted.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: call this when you need local path/hash of collected papers. It explains the conditional behavior of omitting paper_ids, but it does not explicitly mention when to prefer this over sibling tools like search_papers/inspect_papers, nor any exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• collect_papersA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does well: it discloses batch operation, local saving, legal constraints, partial success, and per-item statuses with a specific list of values. This provides substantial behavioral context beyond a vague 'collect papers' claim.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary action, followed by critical constraints and return details. Every sentence adds value with no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (one parameter) and the description covers the main behavioral aspects: what it does, constraints, partial success, and output statuses. It relies on the output schema for detailed return structure, which is appropriate. Minor gaps include no mention of storage location or how saved files are retrieved, but sibling get_collected_papers likely covers that.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage for the single 'paper_ids' parameter. While the description does not add parameter-specific details, the parameter name is self-explanatory (IDs of papers to collect) and the tool purpose makes its meaning clear. The lack of elaboration is acceptable but not exemplary.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly identifies the tool's action (batch-fetch and locally save full-text PDFs) and resource (multiple papers' full-text PDFs). It distinguishes itself from sibling tools like search_papers, inspect_papers, and get_collected_papers by focusing on the downloading/saving action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for collecting PDFs but does not explicitly state when to choose this tool over alternatives. It notes legal constraints (only publicly/legally obtainable texts) but lacks clear 'when-to-use' or 'when-not-to-use' guidance relative to siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspect_papersA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly states that no files are saved and enumerates the possible fulltext.status values (available, metadata_only, restricted, not_found, unknown), giving the agent concrete knowledge of what to expect from the tool's output.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three short sentences, each adding a distinct piece of information: purpose, input source and side effect disclaimer, and output status vocabulary. There is no redundancy or filler content; every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity (single parameter, no annotations) and the presence of an output schema, the description is complete: it explains the input origin, the operation, and the meaning of status values. An agent has all necessary context to invoke the tool correctly and interpret the response.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only defines paper_ids as an array of strings, with 0% description coverage. The description compensates by explaining that these IDs come from search_papers, adding critical semantic context about the parameter's origin and intended use. It does not provide format examples, but the schema already covers the type structure.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states 'Investigate detailed metadata and full-text availability for multiple papers at once,' which clearly identifies the tool's function and resource. It distinguishes itself from siblings: search_papers finds papers, collect_papers saves them, and get_collected_papers retrieves saved ones, while this tool inspects details without saving.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage context: 'Pass paper_id returned by search_papers,' establishing a clear workflow after searching. It also warns 'Does not save files,' which implicitly differentiates it from collect_papers and tells the agent when not to use this tool for saving purposes.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.