Server Quality Checklist

Tool Scores

• list_customer_vaultA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must carry the behavioral disclosure burden. It mentions 'Read-only' and optional date-range filtering, which are useful, but it does not disclose pagination, return format, permissions, or limitations. There is no contradiction with annotations, but the disclosure is minimal.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence, front-loaded with the action and resource. It is concise, but the parenthetical 'report_type=customer_vault' introduces a technical detail that is not explained and may be unnecessary for the agent.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with three optional parameters and no output schema, so the description should explain what the tool returns. It covers purpose and filtering options but does not describe the response shape or any caveats, leaving the agent to infer the return format from the tool name.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% coverage for parameter descriptions, so the baseline is met. The description adds a general statement about optional filtering, but it does not provide additional semantic detail beyond what is already in the schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states that this tool lists stored Customer Vault records, with an optional date-range filter. This distinguishes it from sibling tools like get_customer_vault_record, which retrieves a single record, by focusing on the list action.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context that this is for listing records with optional filtering, but it does not explicitly state when to use this tool versus alternatives such as get_customer_vault_record or search_transactions. Usage is implied rather than explicitly directed.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_subscriptionsA
  3.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of safety disclosure. It explicitly says 'Read-only' and mentions filtering by date range, which adds behavioral context. However, the unexplained '(report_type=recurring)' is confusing and may mislead, and no pagination or response details are given.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short and front-loaded, but the parenthetical '(report_type=recurring)' is an unnecessary implementation detail that doesn't earn its place and obscures clarity. The structure is otherwise efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with no output schema, the description covers purpose, filtering, and read-only behavior. However, it omits common list-operation details like pagination, response format, or preconditions, and the confusing report_type mention reduces completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with each parameter described, so the schema does the heavy lifting. The description adds a general 'created/updated date range' hint but provides no additional syntax or format details beyond what's in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'List recurring subscriptions' with optional date range filtering. This distinguishes it from siblings like get_subscription (single) and list_recurring_plans (plans), using a specific verb and resource.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for listing recurring subscriptions but does not explicitly state when to use this tool over alternatives. It lacks exclusions or references to sibling tools, so guidance is implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_transactionsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses that the tool is read-only, which is a useful safety trait given no annotations exist. However, it does not explain the return format, whether results are a list, how filters combine, or any pagination/error behavior, leaving some behavioral ambiguity.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One concise sentence immediately conveys the tool's purpose and filter options, with no wasted words. It is front-loaded and efficiently structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With 8 parameters and no output schema, the description is too brief to be complete. It doesn't clarify that start_date and end_date are optional despite the 'date range' phrasing, nor does it explain search result behavior, any constraints, or how filters interact, creating ambiguity for a complex tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides descriptions for all 8 parameters, so the description adds little beyond enumerating filter types. The mapping from 'payment type' to transaction_type is implicit but not harmful. Baseline 3 is appropriate when schema coverage is 100%.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches transactions by date range and optional filters, which is a specific verb+resource+scope. It distinguishes itself from sibling tools that handle recurring plans, subscriptions, and customer vault records by focusing on transactions.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context as a search tool with read-only behavior, implying it is for querying transactions rather than modifying them. However, it does not explicitly mention when to use this instead of get_transaction or other alternatives, nor does it state exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_transactionA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses 'Read-only' which is a useful safety trait, but does not mention error behavior, return format, or rate limits. This is acceptable for a simple read operation but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two short, front-loaded sentences with no redundant words. Every clause contributes to understanding the tool's purpose and safety, making it highly efficient.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no output schema) and the presence of siblings, the description covers intent and read-only nature sufficiently. Some details like return value or error cases are omitted, but they are not critical for a basic fetch operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description covers 100% of the parameter, already indicating it is the gateway transaction ID. The description adds no new semantics beyond what the schema provides, thus baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Fetch' with a specific resource ('a single transaction') and the required identifier ('by its gateway transaction ID'). It distinguishes from sibling tools like search_transactions and list_* tools by emphasizing 'single' and the gateway ID lookup.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when you have a gateway transaction ID and need one specific transaction. It does not explicitly mention alternatives or exclusions, but the context is clear enough to infer appropriate use.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_recurring_plansA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description explicitly states 'Read-only,' which is an important behavioral trait for an agent to know. It also mentions optional filtering, but does not disclose limits, pagination, or error behavior. Still, the read-only declaration adds meaningful context beyond the bare listing.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with a parenthetical, no redundant content, and front-loads the core action. Every word earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list tool with one optional parameter and no output schema, the description covers the essential context: what it lists, the optional filter, and that it is read-only. It does not mention pagination or return format, but these are not critical for this simple use case.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of parameters, with plan_id already described as optional and returning a single plan. The description repeats this via 'optionally filtered to a single plan.' Since the schema is fully documented, the description adds minimal extra parameter semantics but is not deficient.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('List') and resource ('recurring billing plans'), and clearly distinguishes from siblings like list_subscriptions or list_customer_vault by naming a unique resource type. The optional filter to a single plan is also stated.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use the tool (to list recurring billing plans) but does not compare it to alternatives such as list_subscriptions or get_subscription. No explicit exclusions or 'when not to use' guidance is provided.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_customer_vault_recordA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It explicitly discloses 'Read-only' and adds a critical safety detail: 'Returns stored profile data only — never full card numbers.' This goes beyond a mere fetch statement, though it does not mention error behavior or auth requirements, which are minor for a read operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences, front-loaded with the core action, followed by useful privacy and safety context ('never full card numbers') and a clear 'Read-only' declaration. Every word earns its place; no filler or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter read-only tool without an output schema, the description explains the tool's scope (single record), the expected input (customer_vault_id), and key behavioral constraints (read-only, no full card numbers). It is sufficient for an agent to invoke correctly, though it omits potential edge cases like 'not found' responses, which are not essential for basic usage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%: the parameter 'customer_vault_id' is fully described as 'The Customer Vault ID to look up.' The description repeats the parameter name but adds no extra semantic detail (e.g., format, length constraints beyond minLength). Baseline 3 is appropriate because the schema already handles parameter meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Fetch a single stored Customer Vault record'), the resource ('Customer Vault record'), and the key identifier ('customer_vault_id'). It also disambiguates from sibling tools like list_customer_vault by specifying 'single', and from get_transaction/get_subscription by naming the vault-specific report_type.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'by its customer_vault_id' makes clear this tool is for looking up a specific vault record when the ID is known, providing implied usage context. It does not explicitly name alternatives or exclusions, but the presence of sibling list tools suggests the contrast is unnecessary for this simple read-only fetch.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_subscriptionA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly states 'Read-only', which is a key safety trait. It does not cover errors or authentication, but for a simple fetch operation this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that delivers the essential information: what it does, how, and the read-only nature. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple single-fetch tool with one well-documented parameter and no output schema, the description covers the purpose, input, and safety. It could be slightly more explicit about the output, but 'Fetch a single recurring subscription' implies the return value, making it sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers 100% of the parameter, and the parameter description already explains 'The recurring subscription ID to look up'. The tool description merely repeats this with 'by its subscription ID', adding no new semantic value beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'Fetch a single recurring subscription by its subscription ID', using a specific verb and resource. It distinguishes itself from sibling tools like get_transaction and list_subscriptions by targeting a single recurring subscription.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear context for when to use the tool: when a specific recurring subscription ID is known. It does not explicitly mention alternatives or exclusions, but the 'report_type=recurring' hint and the singular nature imply when it is appropriate compared to list_subscriptions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.