Server Quality Checklist

Tool Scores

• update_itemsC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions that 'TableId and Id cannot be updated' and 'HTML included in the request values must be contained within CDATA tags,' which are useful behavioral constraints. However, it lacks details on permissions, error handling, rate limits, or the impact of updates (e.g., whether changes are reversible). For a mutation tool with zero annotation coverage, this is insufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections: a purpose statement, key constraints, and parameter/return explanations. It's appropriately sized and front-loaded with essential information. However, it could be more concise by integrating the constraints into the main flow rather than as separate bullet points, and some sentences are slightly verbose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (a mutation tool with 1 parameter but a rich nested schema) and the presence of an output schema (which covers return values), the description is moderately complete. It explains the purpose, key constraints, and parameter basics, but lacks behavioral details like permissions or error handling. With no annotations and incomplete parameter coverage, it should do more to be fully helpful.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description includes an 'Args' section that explains the 'items' parameter as a 'List of item dictionaries to update' with a required 'id' field. However, the input schema has 0% description coverage, meaning all parameter details are undocumented in the schema. The description adds some value by clarifying the structure and the immutable 'id', but it doesn't cover the many other fields in the CatalogItem schema (e.g., name, price, table_id, etc.). Given the low schema coverage, the description partially compensates but not fully.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Update items in Turbify Merchant Solutions catalog using the Catalog API.' It specifies the verb ('update'), resource ('items'), and system context ('Turbify Merchant Solutions catalog using the Catalog API'). However, it doesn't explicitly differentiate from sibling tools like 'create_items' or 'delete_items' beyond the update action, which is why it doesn't reach a score of 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides minimal usage guidance. It mentions that 'TableId and Id cannot be updated,' which is a constraint, but it doesn't explain when to use this tool versus alternatives like 'create_items' or 'delete_items,' nor does it discuss prerequisites or typical scenarios. Without explicit when-to-use or when-not-to-use guidance, the score is limited.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• advanced_search_catalog_itemsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions pagination defaults and a max value for end_index, which adds some context, but it doesn't cover critical aspects like rate limits, authentication needs, error handling, or what 'JSON string with search results' entails. For a search tool with 5 parameters and no annotations, this leaves significant gaps in understanding behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded: it starts with a clear purpose statement, then lists parameters with brief explanations and examples, and ends with return info. Every sentence earns its place without redundancy, making it efficient and easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 5 parameters, 0% schema coverage, no annotations, but an output schema exists, the description is moderately complete. It covers parameter semantics well and notes the return format, but lacks behavioral context (e.g., performance, errors) and sibling differentiation. The output schema reduces the need to explain return values, but overall completeness is adequate with clear gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds substantial meaning by explaining each parameter's purpose, providing an example for criteria_list, clarifying match_type options and defaults, and noting constraints like max for end_index. This goes well beyond the bare schema, though it could detail attribute/operator values or table_id usage more.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool performs 'Advanced search for items in the Turbify Store catalog with multiple criteria,' which specifies the verb (search), resource (catalog items), and scope (advanced with multiple criteria). It distinguishes from simpler search_catalog_items by emphasizing 'advanced' and 'multiple criteria,' though it doesn't explicitly contrast with get_catalog_items or other siblings beyond this implication.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like search_catalog_items or get_catalog_items. It mentions 'advanced search' but doesn't clarify scenarios where this is preferred over simpler methods, nor does it address prerequisites or exclusions. Usage is implied through the term 'advanced,' but explicit context is lacking.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• delete_itemsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It states the tool deletes items but lacks critical behavioral details: whether deletions are permanent or reversible, if there are rate limits, what permissions are required, or how errors are handled. The mention of 'single operation' hints at batch behavior but is vague. This is inadequate for a destructive operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and front-loaded: the first sentence states the core purpose, followed by structured Args and Returns sections. Every sentence adds value without redundancy, making it efficient and easy to parse.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (destructive batch operation), lack of annotations, and an output schema (which covers return values), the description is minimally complete. It explains what the tool does and the parameter, but misses critical context like safety warnings, error handling, or usage scenarios. The output schema reduces the burden, but gaps remain for a mutation tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It adds meaning by explaining 'item_ids' as 'List of item IDs to delete', clarifying the parameter's purpose beyond the schema's type definition. However, it doesn't specify ID format, constraints, or examples. With 0% coverage and one parameter, this earns a 4 for adding basic semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Delete multiple items in a single operation.' This specifies the verb ('Delete') and resource ('items'), and distinguishes it from siblings like 'create_items' or 'update_items'. However, it doesn't explicitly differentiate from potential deletion alternatives (none listed in siblings), keeping it at 4 rather than 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., permissions), exclusions, or comparisons to other tools like 'update_items' for modifying instead of deleting. The agent must infer usage from the name and context alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_catalog_itemsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the tool retrieves details but doesn't mention whether this is a read-only operation, if it requires authentication, potential rate limits, error conditions, or how it handles invalid IDs. This leaves significant gaps in understanding the tool's behavior and constraints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and extremely concise. It uses a clear header format with 'Args' and 'Returns' sections, contains no redundant information, and every sentence directly contributes to understanding the tool's functionality without any wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (which handles return value documentation) and only one parameter, the description covers the basics adequately. However, as a retrieval tool with sibling alternatives and no annotations, it should ideally provide more context about when to use it versus other search tools and any behavioral constraints.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description coverage is 0%, so the description must compensate. It clearly explains that 'item_ids' is a 'List of item IDs to retrieve', adding essential meaning beyond the bare schema. However, it doesn't provide examples of valid ID formats, constraints on list size, or whether IDs must exist, leaving some ambiguity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with a specific verb ('Get details') and resource ('multiple catalog items'), making it immediately understandable. However, it doesn't explicitly distinguish this tool from its sibling 'search_catalog_items' or 'advanced_search_catalog_items', which likely have overlapping functionality for retrieving catalog items.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'search_catalog_items' or 'advanced_search_catalog_items'. It mentions retrieving items by IDs but doesn't specify prerequisites, limitations, or typical use cases, leaving the agent with insufficient context for optimal tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_itemsB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden for behavioral disclosure. It mentions that 'HTML included in the request values must be contained within CDATA tags', which is useful technical context. However, it doesn't address critical behavioral aspects like authentication requirements, rate limits, error handling, whether creation is idempotent, or what happens with duplicate IDs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections: purpose statement, technical requirement, parameter documentation, and return value. Each sentence adds value, though the parameter documentation could be more concise given the extensive schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a creation tool with no annotations and a complex schema, the description provides basic purpose and parameter guidance but lacks important context. It doesn't explain the return format beyond 'JSON string with creation results', though the output schema exists. More behavioral context would be helpful given the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description must compensate. It provides detailed semantic information about the 'items' parameter, listing 6 required fields and 1 optional field with their types and meanings. This adds significant value beyond the bare schema, though it doesn't cover all 40+ fields in the CatalogItem definition.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Create items'), target resource ('Turbify Merchant Solutions catalog'), and method ('using the Catalog API'). It distinguishes this from sibling tools like 'update_items' or 'delete_items' by specifying creation rather than modification or deletion.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'update_items' or 'get_catalog_items'. It mentions the Catalog API context but doesn't specify prerequisites, constraints, or typical use cases for creation versus other operations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_catalog_itemsA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes the search behavior (keyword matching against specific fields), pagination behavior with defaults and limits, and return format ('JSON string with search results'). It doesn't mention rate limits, authentication requirements, or error conditions, but provides solid operational context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently structured with a clear purpose statement, usage guidance, parameter documentation, and return information - all in just 7 lines. Every sentence adds value, and the information is front-loaded with the most important details first.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (search with pagination), no annotations, and the presence of an output schema, the description provides strong contextual coverage. It explains the search scope, parameter semantics, pagination behavior, and return format. The main gap is lack of information about authentication, rate limits, or error handling, but the output schema reduces the need to describe return values in detail.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description fully compensates by explaining all three parameters: 'keyword' (what it matches against), 'start_index' (pagination starting point with default), and 'end_index' (pagination ending point with default and max). The description adds meaningful context beyond what the bare schema provides about parameter purposes and constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Search for items') and resource ('Turbify Store catalog'), and explicitly distinguishes this tool from its sibling 'advanced_search_catalog_items' by noting this is a 'simple search' with limited field matching. This provides excellent differentiation from alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use this tool ('simple search only matches keyword against item ID, name, or code fields') and when to use an alternative ('For more advanced search capabilities against other fields, use advanced_search_catalog_items'). This gives clear context for tool selection among siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.