Server Quality Checklist

Tool Scores

• compare_vendorsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate read-only and open-world behavior. The description adds that each SLA term includes its source, which is useful but limited. No mention of data freshness or vendor coverage constraints.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single clear sentence with no wasted words. It efficiently conveys the tool's purpose and key output details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple parameter set, full schema coverage, and annotations, the description is largely sufficient. It lacks output format details but remains adequate for the tool's straightforward function.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers the parameter 'vendors' with 100% description. The description does not add extra meaning beyond the schema (e.g., no format details or case sensitivity hints).

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'compare', the resource 'vendors', and the specific SLA terms (uptime, max credit, etc.). It distinguishes from sibling tools like get_service_sla which focuses on a single vendor/service.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use when comparing multiple vendors' SLAs, but does not explicitly state when to use versus alternatives like search_vendors or vendors_in_category. No exclusion criteria are provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_vendorsA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations declare readOnlyHint and openWorldHint. Description adds behavior: 'Returns slugs to look up,' indicating what the output contains. No contradictions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, no fluff. All information is relevant and presented efficiently.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple search tool with one parameter and good annotations, description covers purpose, query examples, and output type. Lacks mention of result limits or case sensitivity, but sufficient.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with description 'Free-text: a vendor name, service, or category.' Description adds concrete examples ('datadog', 'storage', 'payments') which enhance understanding beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description clearly states it searches SLA.directory for vendors and services by name or category, and returns slugs. It is distinct from sibling tools like get_vendor or compare_vendors, but does not explicitly differentiate.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides examples of queries but does not specify when to use this tool over alternatives like get_vendor or compare_vendors. No explicit when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• vendors_in_categoryA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare 'readOnlyHint' and 'openWorldHint', describing safety and open-world nature. The description adds minimal extra behavior ('vendor summaries') but does not disclose pagination, rate limits, or other traits. With annotations covering the core profile, a 3 is appropriate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single sentence with a parenthetical example and cross-reference to 'list_categories'. No extraneous content; every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with one parameter, no output schema, and safety annotations, the description is sufficient. It explains the purpose, parameter source, and an example. Minor gap: does not clarify if results are paginated or limited, but not critical given the tool's simplicity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the schema description for 'category' is 'Category slug.' The description repeats the example slug but adds no new semantic info beyond what the schema provides. Baseline 3 for full schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List vendor summaries') and the resource ('in a category'), with a concrete example ('e.g. 'finance-and-payments''). It also references 'list_categories' to help obtain the slug, distinguishing it from siblings like 'search_vendors' or 'get_vendor'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly directs users to use 'list_categories' to obtain the correct category slug, providing context on how to use this tool. While it does not explicitly state when not to use it, the specificity of listing vendors in a category implies its niche compared to broader tools like 'search_vendors'.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_categoriesA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations provide readOnlyHint and openWorldHint. Description adds that result includes vendor counts, but no detail on behavior (e.g., pagination, limits). Adequate but not enhanced beyond annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded with key info. Zero redundancy. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no parameters, rich annotations, and no output schema, the description sufficiently explains purpose and result. No missing elements for this simple list tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  No parameters exist (schema coverage 100%). Description adds nothing about params, but none needed. Baseline for zero parameters is 4.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Clearly states 'List all SLA.directory categories with vendor counts.' Verb 'list' and resource 'categories' are specific. Distinct from sibling tools like vendors_in_category or search_vendors, which focus on vendors.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Implies use when categories with counts are needed, but no explicit guidance on when not to use or alternatives. Sibling tools exist for vendor-specific queries, but description does not differentiate usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_service_slaA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate readOnlyHint and openWorldHint. The description adds context about multiple SLAs per vendor but doesn't elaborate on behavior like rate limits or auth. No contradiction with annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences, front-loaded with the verb and resource. No unnecessary words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter tool with no output schema, the description adequately covers purpose and optional filtering. Could mention return type but not required.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with clear descriptions. The description adds value by explaining the rationale for the optional service filter ('many platforms publish a separate SLA per service').

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Get' and identifies the resource 'per-service SLAs for a vendor'. It naturally distinguishes from sibling tools like 'get_vendor' or 'compare_vendors' by focusing on SLAs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description clearly states the tool's purpose and mentions optional filtering. While it doesn't explicitly list when not to use it, the context from sibling tools implies its specific use case for SLA retrieval.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_vendorA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description reveals key behaviors: fuzzy matching on vendor name, default summary vs raw format. Annotations confirm read-only and open-world intention. No contradictions, though additional details on edge cases would be beneficial.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loading the main action and then detailing input/output. Every word is necessary; no redundancy. Highly efficient for agent consumption.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, the description explains the returned format ('cited summary' vs 'raw'), but does not detail the structure of the summary. Given the tool simplicity and clear purpose, it is adequate but could describe the summary fields for full completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, but the description adds value beyond the schema by clarifying the default format and providing concrete examples ('aws', 'Amazon Web Services'). This helps the agent understand parameter usage without referencing the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool retrieves a vendor's SLA, listing specific details (uptime, credit policy, claim process, per-service SLAs). It distinguishes from siblings like compare_vendors and get_service_sla by focusing on a single vendor's aggregated SLA information.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explains inputs (slug or name, fuzzy) and output format options, helping the agent decide when to use this tool. It does not explicitly state when not to use it, but the sibling list provides context for differentiation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.