Server Quality Checklist

Tool Scores

• list_libraryA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries full burden. It discloses that each paper line includes its item key in [KEY] brackets and that this key is used in other tools — a useful behavioral detail. However, it does not mention pagination, sorting, exact output scope, or whether the list includes collections only as names.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the primary action, and the second sentence provides crucially useful downstream context about item keys. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (not shown) and a small sibling set, the description is largely complete: it covers the main list functionality, optional filters, and the connection to paper_details/paper_text via the item key. It lacks a few edge details like default sort order, but is sufficient for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema only defines a nullable string 'query' with 0% coverage, so the description must compensate. It explains that the parameter filters by collection name, paper title, or author, but is vague about the exact format — whether it's a single free-text search or separate fields. It adds some meaning but not enough for precise invocation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('List') and resource ('collections and papers in the Zotero library'), and it distinguishes itself from siblings: paper_details and paper_text focus on individual items, while search_papers implies searching rather than listing all.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It says the list can be optionally filtered by collection name, paper title, or author, which indicates a browsing/listing use case. It also points to paper_details and paper_text for downstream access, but does not explicitly state when not to use this tool versus search_papers.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• paper_textA
  4.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations provided, so the description carries the full burden. It does not disclose side effects, performance implications, error handling, or explicitly confirm it is read-only. Mentioning 'using pdftotext' is an implementation detail, not behavioral transparency. The extract action implies reading, but the description lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no filler. The first sentence states the action, and the second provides essential input sourcing. It is well-structured and front-loaded with the primary purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, output schema present), the description covers purpose and parameter sourcing adequately. It does not mention limitations like scanned PDFs or potential errors, but the presence of an output schema likely handles return values. Slight gap in edge-case behaviors.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With a single parameter and 0% schema description coverage, the description fully compensates by explaining item_key as a paper identifier and providing specific instructions on where to find it (list_library output prefixed with [KEY], or search_papers results). This adds meaning beyond the bare string type in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool extracts full text from a paper's PDF, using the specific verb 'Extract' and identifying the resource. This distinguishes it from siblings like list_library (listing papers), paper_details (metadata), and search_papers (searching), as it's the only one that retrieves full text content.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context by instructing how to obtain the required item_key from list_library or search_papers results. This implies the tool is used when full text is needed and even mentions prerequisite tools, though it does not explicitly contrast with alternatives or provide exclusion criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_papersA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of transparency. It discloses the result structure and the need for keys, but it does not mention matching behavior (exact vs. fuzzy), pagination, result limits, or error conditions, leaving gaps.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the action, uses three concise sentences, and includes no redundant details—every sentence serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema covers return values, the description sufficiently covers purpose, result fields, and downstream tool usage. A brief mention of how it differs from list_library would make it more complete, but it is adequate as-is.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides zero coverage for the single 'query' parameter, but the description clarifies that it is a title/author search term, adding semantic meaning beyond the bare schema definition.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Search papers by title or author') and the specific resource, and differentiates from siblings by mentioning the returned fields and how to use the key with paper_details or paper_text.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides clear context for when to use the tool (searching by title/author) and explains the follow-up usage of keys with related tools, though it does not explicitly mention when not to use it or contrast with list_library.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• paper_detailsA
  4.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of disclosing behavior. It communicates that the operation is a read ('Get') and focuses on metadata, which is the primary behavior. However, it does not disclose potential error conditions, permission requirements, or any side effects, leaving some room for ambiguity in edge cases.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two clear sentences. The first sentence immediately states the tool's purpose, and the second provides necessary usage guidance. There is no redundant information or filler, making it exceptionally concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has only one parameter and an output schema exists, the description covers all necessary context: the purpose, the source of the key, and the fact that the output is full metadata. The output schema handles return value specifics, so the description is complete for this low-complexity tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must add meaning to the parameter. It does this effectively by explaining what item_key is ('shown as [KEY]') and exactly where to get it from (list_library or search_papers). This is far more informative than the schema's bare 'string' type, fully compensating for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with 'Get full metadata for a paper', clearly specifying the action (get) and the resource (paper metadata). It distinguishes itself from siblings like list_library and paper_text by focusing on metadata retrieval. It also mentions how to obtain the required item_key, reinforcing the tool's specific role.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states where the item_key comes from ('Obtain the item_key from list_library output... or from search_papers results'), which gives clear context for when to use this tool. It doesn't explicitly exclude alternatives, but the reference to source tools implies a workflow. No alternatives are named directly, so it's not a full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.