Server Quality Checklist

Tool Scores

• library_listA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must convey behavior alone. It clarifies the scope (personal library) and optional filtering, but does not disclose return format, ordering, or explicitly confirm read-only behavior. For a list tool this is moderate but leaves some gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence followed by concrete examples, with no redundant information. It is front-loaded with the verb and resource, making it highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with optional filters, the description adequately covers purpose, scope, and usage contexts. The schema handles all parameter details. While there is no output schema, the tool's simplicity makes this acceptable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with all four parameters described in detail. The description's mention of 'optionally filtered' adds no new semantics beyond what the schema already provides, so the baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists the user's personal arXivSub library, including collections and saved items, and distinguishes it from sibling search/save tools by emphasizing the personal library scope. The use-case examples further clarify its purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides specific example queries ('what have I saved', 'show my library') that indicate when to use the tool. It does not explicitly mention alternatives or exclusions, but the intended usage is clear from these examples.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• find_similar_papersA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of disclosing behavior. It explains the operation is read-only in effect, returns "the most similar papers... each with the full summary and code link," and notes the seed must come from a prior search result. It does not discuss error handling or edge cases, but for a retrieval tool this is sufficient; the disclosure is clear and non-misleading.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences long, with the core purpose stated first, followed by usage guidance and a chaining example. Every sentence adds value, there is no repetition of schema content, and the structure is well-organized and front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no output schema, the description tells the user what to expect (full summary and code link) and how to use the tool iteratively. It covers prerequisites (a paper from a prior search), the use cases, and the workflow with search. It does not mention potential error scenarios, but for a straightforward recommendation tool, the description is sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% coverage of all four parameters, so the description does not need to add parameter-level detail. The description reinforces that the seed paper must already exist ("from a prior search result"), which aligns with the schema's emphasis on using a real arXivSub id. Since schema coverage is complete, the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific, actionable statement: "Given ONE paper you already have... return the most similar papers from the same corpus — semantic nearest neighbours." This clearly identifies the tool's function and resource, and distinguishes it from sibling search or library tools by emphasizing that it requires a prior paper as input rather than a query.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states common use cases: "Use for 'more like this', 'related work', 'what else is in this line of work', or to expand a literature review." It also provides a chaining workflow (search -> pick a seed -> find_similar_papers -> repeat), which gives practical guidance. However, it does not explicitly name when NOT to use this tool or mention specific alternatives, so it falls just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• research_insightA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description takes on the full burden of behavioral transparency. It discloses the output format ('rows of {dim, breakdown, value}') and the access requirement ('Requires Pro or active trial days'), plus a helpful analogy ('same engine as the website's Custom Chart'). It does not mention rate limits or data caveats, but provides solid context for a read-only analytics tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four well-organized sentences cover purpose, use cases, output, and a prerequisite. The description is front-loaded with the core function and uses compact, meaningful examples. No filler or redundant restatement of schema fields.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having 6 parameters and no output schema, the description gives enough context for an agent to invoke the tool correctly: it explains what the tool does, when to use it, what the result looks like, and the Pro requirement. The schema handles parameter details, and the description fills the remaining conceptual gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% parameter coverage with descriptions and enums, so the baseline is 3. The description adds a few clarifying examples and output shape but does not substantially expand on the schema's parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific action ('Build aggregate pivot statistics') and a clear resource ('arXivSub corpora'), immediately distinguishing it from sibling search tools. It also provides concrete example queries ('top institutions in RL', 'papers per month by keyword') that confirm its analytical purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly states when to use the tool: for trends, rankings, breakdowns, and comparisons, with illustrative examples. It does not explicitly discuss when not to use it or name alternatives, but the intended use case is unambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_papersA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It explains the search mechanism ('semantic + keyword search over a structured, summarized corpus') and the return format ('6-part summary', 'authors + affiliations', 'pdf_url', 'github_url'), adding valuable context beyond the schema without contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: a clear purpose sentence, a usage sentence, and a return-value sentence. No filler or redundancy, with the most important information front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite having no output schema, the description gives a thorough explanation of what is returned (6-part summary, authors, affiliations, URLs). Combined with the use-case guidance and clear search scope, it fully covers the tool's behavior for an agent in this complex domain.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema descriptions cover 100% of the 7 parameters, so the description doesn't need to add parameter-level detail. The description does not provide additional semantic meaning beyond the schema; it focuses on the overall tool behavior, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Search' with a clearly defined resource: 'academic papers from arXiv and major AI/CV/ML/robotics/NLP conferences'. This distinguishes it from sibling tools like find_similar_papers (similarity search), research_insight (insights), and library_save/list (library operations).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool: 'for "find papers on X", "latest research about Y", "recent conference work on Z", or to start a literature review'. It provides clear context for use, but does not mention when not to use it or name alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• library_saveA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden. It discloses key behaviors: idempotency ('saving the same paper again updates only the fields you pass and never duplicates') and the deletion limitation. These are important side effects not inferable from the name or schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose, then examples, then key behavioral caveats. Every sentence earns its place with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a 5-param tool with no output schema, the description covers purpose, usage, idempotency, and deletion constraints. It does not mention authentication or return values, but these are not essential for tool selection. The lack of explicit reference to the item_id coming from a prior search is compensated by the schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already documents all 5 parameters with descriptions, so baseline is 3. The description's natural-language examples ('note that', 'mark as read') map to note and reading_status, but add little new semantic detail beyond the schema. The partial-update note is behavioral rather than parameter-specific.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'Save (or update) a paper in the user's personal arXivSub library', using a specific verb+resource. It clearly distinguishes from siblings like search_papers and library_list by focusing on persistence into the user's library. The example intents further pin down the purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit usage phrases are given: 'save this', 'bookmark that paper', 'add it to my <X> collection', 'note that ...', 'mark as read'. It also states a clear non-use case: 'Cannot delete — removals are done on the website.' This provides strong when-to-use guidance without needing to name alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.