Server Quality Checklist

Tool Scores

• search_library_papersA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare read-only, idempotent, non-destructive behavior; the description adds meaningful context beyond that by specifying the return list includes title/abstract and source, and by mentioning indexed-conference coverage. It could disclose pagination limits or result-count behavior, but the key traits are covered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the purpose, followed by a coverage caveat, return summary, and a structured Args list. It is slightly padded by 'Indexed conferences: none,' but the sentence is informative and not redundant.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool without an output schema, the description explains the return shape, source inclusion, display modes, ordering, and pagination. It is not fully complete because page size and precise matching behavior (e.g., metadata vs full-text) are unspecified, but the main workflow is clear.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has zero descriptions, so the Args section carries the full explanatory burden and does so thoroughly: each parameter is named, with defaults, allowed values, and the page/start_from mutual-exclusivity constraint.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Search the local paper database by keywords.' It clarifies the local scope, distinguishing it from sibling search_arxiv_papers, and the return of paper lists with abstracts differentiates it from grep_paper_content/get_paper_toc.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this is for searching the local paper database, and 'Indexed conferences: none' provides a context cue, but it never explicitly states when to prefer this over sibling tools or gives exclusion criteria. No alternatives are named.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_arxiv_papersA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnly, idempotent, and non-destructive behavior. The description adds valuable context: output includes title/abstract and submit date, and it explains parameter behaviors like the mutual exclusivity of 'page' and 'start_from'. This goes beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description front-loads the purpose in two sentences, then systematically lists parameters in a clear structure. Every sentence adds value, especially given the need to compensate for the absent schema descriptions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 10-parameter search tool, the description covers output format, parameter semantics, and special constraints. It lacks examples or error behavior, but the rich parameter detail and annotations make it reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description's 'Args' section thoroughly documents all 10 parameters, including their meanings, defaults, formats, and mutual exclusivity. This fully compensates for the missing schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Search'), names the resource ('arxiv'), and clearly states the return value ('Return paper list with title (and abstract)'). The tool name itself distinguishes it from the sibling 'search_library_papers'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear context: search arxiv for latest papers. It implicitly differentiates from sibling tools by specifying 'arxiv', but does not explicitly mention alternatives or when-not-to-use. This fits 'clear context, no exclusions'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• grep_paper_contentA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare this as a safe, read-only, idempotent operation, so the bar is lower. The description adds useful behavioral detail: it pipes markdown as stdin, returns stdout, enforces a max_chars limit, and explains fuzzy title matching. This enriches the annotation-only picture without contradicting it.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured: a one-sentence purpose statement, a clear return line, and then parameter explanations with call-worthy examples. Every sentence earns its place without fluff, making it both concise and informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity and lack of output schema, the description covers all essential aspects: how it works (stdin piping), what it returns (stdout), parameters, usage tips, and output limits. The note to call get_paper_toc first completes the workflow guidance. It is complete for this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description carries the full burden. It explains each parameter thoroughly: title with fuzzy matching, grep_command with multiple examples including a recommendation to use the TOC first, and max_chars with its default. This fully compensates for the empty schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool greps patterns in a paper's full markdown text by piping it to a user-supplied grep command. This specific verb+resource clearly distinguishes it from sibling tools like search_library_papers or get_paper_toc, which operate at a higher level.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear practical context with concrete examples (reading sections, finding context) and advises calling get_paper_toc first for exact section names. However, it does not explicitly name alternative tools or state when not to use this tool, so it falls short of the highest level of guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_paper_tocA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only and idempotent behavior. The description adds useful context beyond annotations: fuzzy title matching and the sequential requirement to call before other content tools. Does not detail return format, but with annotations covering safety, this is a minor gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: first sentence states the purpose, second provides usage order, and third explains the parameter. No unnecessary fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter, read-only tool, the description covers purpose, usage sequence, and parameter semantics adequately. It does not describe the return format, but given the simple nature and existing annotations, the description is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Input schema only defines title as a string. The description explains the meaning of the title parameter, notes that fuzzy matching is supported, and advises providing it as accurately as possible. This fully compensates for the 0% schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves the table of contents of a paper. It uses a specific verb ('Get') and resource ('table of contents'), and distinguishes itself from sibling tools like grep_paper_content by positioning itself as the first step before content reading.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly instructs to call this tool FIRST before reading a paper's content, then directs to grep_paper_content for reading chapters or searching. This provides clear when-to-use guidance and names the alternative tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.