Server Quality Checklist

Tool Scores

• get_zotero_informationC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It mentions that returned information can be 'finetuned' by parameters, implying configurability, but lacks details on permissions, rate limits, or what happens if parameters are omitted. For a tool with three parameters and no annotation coverage, this leaves significant behavioral gaps, such as default behaviors or error handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is moderately concise but not optimally structured. The first sentence clearly states the default behavior, but the second sentence ('Good to call if...') is vague and doesn't add value. The third sentence about finetuning is useful but could be more integrated. Overall, it's front-loaded with purpose but includes a redundant phrase that reduces efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has three parameters, no annotations, and an output schema (which reduces the need to describe return values), the description is partially complete. It covers the tool's purpose and parameter finetuning but lacks usage guidelines, behavioral details like authentication or errors, and doesn't fully compensate for the 33% gap in schema description coverage. It's adequate as a baseline but has clear gaps for effective agent use.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds meaningful context beyond the input schema, which has 67% coverage. It explains that the tool returns 'information on the Zotero library' by default and can be 'finetuned' for 'summary, collections, recent items, tags,' clarifying the purpose of the 'properties' parameter. However, it doesn't detail the semantics of 'limit' or 'itemType' beyond what the schema provides, leaving some parameter meaning implicit.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the tool returns information about a Zotero library, which clarifies its purpose. However, it's somewhat vague about what 'information' entails and doesn't clearly differentiate from sibling tools like 'get_collection_items' or 'get_items_metadata' that also retrieve library data. The phrase 'Good to call if you dont have a clear understanding yet what to ask from this tool' adds confusion rather than specificity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides minimal guidance on when to use this tool, only suggesting it's 'Good to call if you dont have a clear understanding yet what to ask from this tool,' which is ambiguous and not actionable. It doesn't explain when to choose this tool over alternatives like 'search_library' or how it complements sibling tools, leaving the agent without clear usage criteria.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_collection_itemsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states the action ('Gets') but doesn't cover critical aspects like whether this is a read-only operation, potential rate limits, authentication needs, or how it handles errors. The description is too minimal for a tool with parameters and an output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with no wasted words. It's front-loaded with the core purpose, making it easy to scan, though this brevity comes at the cost of completeness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has 2 parameters (with 0% schema coverage) and an output schema, the description is incomplete. It doesn't explain parameter meanings or behavioral traits, but the presence of an output schema reduces the need to describe return values. This makes it minimally adequate but with significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate, but it only mentions 'a specific Zotero collection' without explaining what 'collection_key' is or how to obtain it. It doesn't address the 'limit' parameter at all, leaving both parameters poorly documented beyond their schema types.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Gets') and resource ('all items in a specific Zotero collection'), making the purpose explicit. However, it doesn't differentiate from sibling tools like 'get_items_metadata' or 'search_library', which likely retrieve similar data but with different scopes or filters.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, such as needing a valid collection key, or compare it to siblings like 'get_items_metadata' for broader item retrieval or 'search_library' for filtered searches.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_items_metadataC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states this is a 'Get' operation, implying read-only behavior, but doesn't cover critical aspects like authentication needs, rate limits, error handling, or what 'detailed information' includes. For a tool with no annotations, this is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that front-loads the core purpose ('Get detailed information on specific item(s) in the library'). It avoids redundancy and waste, though it could be slightly more structured (e.g., by hinting at parameters).

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (4 parameters, 1 required) and the presence of an output schema (which handles return values), the description is minimally adequate. However, with no annotations and no sibling differentiation, it lacks completeness for safe and effective use. The description should compensate more for the missing annotation context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage, with clear documentation for all parameters (e.g., 'item_key' for retrieval, booleans for content inclusion). The description adds no additional meaning beyond the schema, such as explaining parameter interactions or usage examples. Baseline 3 is appropriate since the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Get') and resource ('detailed information on specific item(s) in the library'), making the purpose understandable. However, it doesn't explicitly distinguish this tool from its siblings like 'get_collection_items' or 'search_library', which likely retrieve items in different ways (e.g., by collection vs. by search).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention siblings like 'get_collection_items' or 'search_library', nor does it specify prerequisites or exclusions (e.g., that 'item_key' is required). This leaves the agent without context for tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_libraryB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure but provides minimal information. It doesn't mention whether this is a read-only operation, what authentication might be required, potential rate limits, or what happens when no results are found. The description only states what the tool does at the highest level without behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is maximally concise - a single sentence that states the core purpose without any fluff. Every word earns its place, and the information is front-loaded. There's no wasted verbiage or unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that there's an output schema (which handles return values), no annotations, and moderate schema description coverage (67%), the description is minimally adequate. However, for a search tool with 6 parameters and sibling alternatives, the description should provide more context about when to use this specific search functionality versus other retrieval methods available in the server.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description provides no parameter information whatsoever, while the schema has 67% description coverage. Since the schema already documents most parameters well (especially 'qmode', 'itemType', and 'tag' with detailed examples), the baseline is 3. The description doesn't add value beyond what's in the schema, but the schema coverage is adequate for most parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Search') and target resource ('Zotero library for an item'), making the purpose immediately understandable. However, it doesn't distinguish this tool from sibling tools like 'get_collection_items' or 'get_items_metadata' - both of which also retrieve items from the library, leaving some ambiguity about when to choose this search tool over those alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance about when to use this tool versus the sibling tools. There's no mention of alternatives, prerequisites, or specific contexts where this search functionality is preferable to 'get_collection_items' or 'get_items_metadata'. The agent must infer usage from the tool name alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.