Server Quality Checklist

Tool Scores

• get_permitteeB
  3.1/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. While it mentions what information is returned, it doesn't address important behavioral aspects like whether this is a read-only operation, authentication requirements, rate limits, error conditions, or what happens with invalid permit numbers. The description provides basic functional information but lacks critical behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is appropriately sized and well-structured with clear sections (purpose, arguments, returns). Each sentence serves a distinct purpose: the first states the tool's function, the second specifies what's returned, and the following sections document parameters and return values. There's minimal wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has an output schema (which handles return value documentation) and only one parameter with good semantic coverage in the description, the description is reasonably complete for a simple lookup tool. However, the lack of behavioral transparency and usage guidelines relative to sibling tools represents a significant gap in contextual understanding for an AI agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage and only one parameter, the description provides essential semantic context that the schema lacks. It explains that 'permit_number' is a 'federal permit number' and provides an example format ('NY-I-123'), which adds meaningful interpretation beyond the bare schema type declaration. This significantly compensates for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get detailed information about a permit holder' with specific details about what information is returned (company details and recent COLAs). It uses a specific verb ('Get') and identifies the resource ('permit holder'), but doesn't explicitly differentiate from sibling tools like 'search_permittees' or 'get_cola'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. With sibling tools like 'search_permittees' and 'get_cola' available, there's no indication of when this specific tool is appropriate versus those alternatives, nor any prerequisites or contextual usage information.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_api_usageA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It adequately describes what the tool returns (usage statistics, rate limits) and implies it's a read-only operation through the verb 'Check.' However, it doesn't mention potential authentication requirements, error conditions, or whether the data is real-time versus cached, leaving some behavioral aspects unclear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and front-loaded, starting with the core purpose and following with details on returns. Every sentence adds value without redundancy, and it efficiently communicates the tool's functionality in a compact format. There's no wasted verbiage or unnecessary elaboration.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (no parameters, with an output schema), the description is largely complete. It explains the purpose and what to expect in returns, which aligns with the presence of an output schema. However, it could be more comprehensive by addressing authentication needs or error handling, slightly reducing completeness for a tool with no annotations.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the input schema has 100% coverage (empty object). The description appropriately doesn't discuss parameters since none exist, focusing instead on the return values. This meets the baseline expectation for a parameterless tool, though it doesn't add extra semantic context beyond what the schema already indicates.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific verb 'Check' and resource 'COLA Cloud API usage and rate limits', making the purpose immediately understandable. It distinguishes this tool from sibling tools like 'get_cola' or 'search_colas' by focusing on API usage metrics rather than data retrieval operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. While it implicitly suggests usage for monitoring API consumption, it doesn't mention prerequisites, timing considerations, or how it relates to other tools in the system. There's no explicit 'when' or 'when not' context provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_colasA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It does well by describing the search functionality, pagination behavior (defaults and max), and return format. However, it doesn't mention important behavioral aspects like rate limits, authentication requirements, error conditions, or whether this is a read-only operation (though 'search' implies it likely is).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections (purpose explanation, Args, Returns) and every sentence adds value. The COLA explanation provides necessary domain context. While not minimal, the length is justified given the 10 parameters needing documentation. The information is front-loaded with the core purpose stated first.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with 10 parameters, 0% schema coverage, no annotations, but with an output schema, the description does an excellent job. It explains the tool's purpose, documents all parameters thoroughly, describes the return format, and provides domain context about COLAs. The main gap is lack of explicit usage guidelines versus sibling tools.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage and 10 parameters, the description fully compensates by providing detailed semantic explanations for every parameter. Each parameter gets clear documentation including search scope ('q: Full-text search query (searches brand, product name, origin)'), format examples ('YYYY-MM-DD'), constraints ('default: 20, max: 100'), and matching behavior ('partial match, case-insensitive').

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose as 'Search and filter COLA (Certificate of Label Approval) records,' providing a specific verb ('search and filter') and resource ('COLA records'). It distinguishes from siblings like 'get_cola' (which likely retrieves a single record) and 'search_permittees' (which searches different entities). The explanation of what COLAs are adds helpful domain context.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage through the explanation of what COLAs are and the search/filter functionality, but doesn't explicitly state when to use this tool versus alternatives. No guidance is provided about when to use 'search_colas' versus 'get_cola' or 'search_permittees,' nor are there any prerequisites or exclusions mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• lookup_barcodeA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses that barcodes are extracted from label images using computer vision, which adds useful context about data source and limitations. However, it doesn't cover behavioral aspects like error handling, rate limits, authentication needs, or what happens if no matches are found.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured and appropriately sized. It front-loads the core purpose, follows with implementation context, and ends with clear sections for Args and Returns. Every sentence earns its place without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (single parameter, no annotations, but with output schema), the description is mostly complete. It explains the purpose, parameter semantics, and return value at a high level. The existence of an output schema means it doesn't need to detail return structure, but could benefit from more behavioral context for a lookup tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds significant meaning beyond the input schema, which has 0% coverage. It explains that barcode_value accepts UPC, EAN formats with an example ('012345678905'), clarifies what constitutes a barcode, and mentions it's extracted from label images. This compensates well for the schema's lack of descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose with specific verbs ('Find', 'identifying', 'tracking') and resources ('COLAs by product barcode'). It distinguishes from siblings by focusing on barcode-based lookup rather than general searches or other entity retrievals like get_cola or search_colas.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use this tool ('useful for identifying products or tracking label changes over time'), but does not explicitly state when not to use it or name specific alternatives among the sibling tools. It implies usage based on barcode availability.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_permitteesA
  4.3/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It discloses that this is a search operation (implied read-only), mentions pagination behavior with defaults, and specifies partial matching for company names. However, it doesn't mention rate limits, authentication requirements, or error conditions that would be helpful for a search tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a clear introductory sentence, domain explanation, and organized parameter documentation. Every sentence earns its place by providing essential information without redundancy. The formatting with clear sections (Args, Returns) enhances readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a search tool with 5 parameters and no annotations, the description does an excellent job explaining parameters and return values. The existence of an output schema means the description doesn't need to detail return structure. It could be slightly more complete by mentioning authentication or rate limits, but overall it's quite comprehensive.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description fully compensates by explaining all 5 parameters in detail. It clarifies that 'q' uses partial matching, 'state' requires two-letter codes, 'is_active' filters by permit status, and provides default values and constraints for pagination parameters. This adds significant value beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches for permit holders (companies authorized to produce/import alcohol) and distinguishes them from other entities like COLAs. It specifies the resource (permit holders/businesses) and verb (search) with domain context about federal permits.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context about what permittees are and their relationship to COLAs, which helps differentiate from sibling tools like 'search_colas' and 'get_permittee'. However, it doesn't explicitly state when to use this versus the 'get_permittee' tool for individual lookups.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_colaA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It effectively describes what the tool returns ('full record including all label images, extracted barcodes, and AI-enriched fields'), which is valuable context beyond just being a read operation. It doesn't mention rate limits, authentication needs, or error conditions, but provides good behavioral insight for a read-only tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is perfectly structured and concise: purpose statement first, return details second, parameter explanation third, and return summary fourth. Every sentence earns its place with no wasted words, and information is front-loaded appropriately.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (retrieving detailed records with images and enriched data), no annotations, and the presence of an output schema, the description provides excellent context. It explains what kind of data is returned, mentions AI-enriched fields, and provides parameter semantics. The output schema will handle return value details, so the description focuses appropriately on usage context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema has 0% description coverage, so the description must fully compensate. It provides excellent parameter semantics: clearly explains what 'ttb_id' represents (TTB ID), provides context about what TTB stands for, and gives a concrete example format ('23001001000001'). This adds substantial meaning beyond the bare schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the specific action ('Get detailed information') and resource ('a specific COLA by its TTB ID'), distinguishing it from sibling tools like search_colas (which likely returns multiple results) and lookup_barcode (which uses a different identifier). It provides a precise verb+resource combination with clear scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use this tool (when you need detailed information about a specific COLA using its TTB ID) versus alternatives like search_colas (which likely searches multiple records). However, it doesn't explicitly state when NOT to use it or name specific alternatives, keeping it at a 4 rather than a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.