Server Quality Checklist

Tool Scores

• get_endpoint_specA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses what the tool returns (parameters, response, error codes), which is the core behavior. However, it doesn't mention any potential error handling, rate limits, or any side effects. As a read-only retrieval, this is acceptable but not exceptionally transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences. The first sentence states the core function, and the second provides usage context. No filler words, and the most critical information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Since there is no output schema, the description correctly explains what the response contains ('parameters, response, error codes'). It covers the essential information for a retrieval tool. The 'section' parameter is described in the schema, so no need to repeat. Slightly more could be said about expected size or format, but it is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description does not add any extra meaning beyond what the schema already documents for 'section' and 'endpoint' – it simply restates the purpose.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Return'), a clear resource ('full reference page'), and the specific content ('parameters, response, error codes'). It clearly distinguishes itself from siblings like search_docs by being the tool that retrieves a specific endpoint's spec.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It provides a clear use case: 'to ground your generated code against the exact request/response schema before emitting it.' This tells the agent when to use it. It doesn't explicitly mention alternatives, but the sibling list includes search_docs, which is likely for broader searching, so the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_test_dataA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must carry the burden of behavioral disclosure. The description implies a read-only data fetch (returning data) but does not explicitly state that it makes no changes or that it is safe to call. It also does not mention error cases, rate limits, or the exact structure of the returned data. Adequate for a simple retrieval, but lacks depth given the absence of annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with zero filler. The primary action and scope are front-loaded, followed by a concise usage recommendation. Every word earns its place; no redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with only 2 parameters, no output schema, and no nested objects, the description covers the essential information: what is returned, its purpose, and how to invoke it. It could optionally describe the response structure more explicitly, but the types of data are listed, which is sufficient for an agent to call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% — both 'country' and 'section' have descriptions, including supported values and defaults. The description adds context about why these parameters matter (to populate realistic sandbox requests) but does not add syntax or format details beyond what the schema already covers. Baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Return' and the resource 'official-shaped sandbox test data for a country', enumerating the exact contents (customers, test cards, PIX/Bre-B keys). This is specific and distinct from sibling tools like get_country_rules or get_endpoint_spec, making it easy for an agent to identify this as the correct tool for generating test data.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Use this to populate sandbox requests with realistic values instead of inventing them', which gives clear context on when to use it. It does not explicitly state when not to use it or compare to alternatives, but the purpose is clear enough that an agent would not confuse it with the sibling tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• search_docsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations are absent, so the description carries the full burden. It discloses the search-and-return behavior (returning matches with snippets) but does not mention permissions, rate limits, or side effects. It's adequate for a search tool but leaves notable gaps in behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two tightly written sentences: the first states the purpose and output, the second gives the usage directive. Every word earns its place, and the key information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a straightforward 3-parameter tool with full schema coverage and no output schema, the description supplies enough context for correct usage, including a strong usage hint. It omits details like pagination or error handling, but these are minor for a search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides 100% coverage with rich descriptions (e.g., query examples, category enum explanations, limit bounds). The description adds little beyond restating the scope, so it meets the baseline for high schema coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific verb ('Search'), a distinct resource ('PayRetailers Guides and Skills'), and the output format ('top matches with a short snippet'). This effectively differentiates it from sibling tools that fetch specific data (e.g., get_endpoint_spec, get_country_rules).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides an explicit 'when to use' directive: 'Use this before answering any integration question to ground your response in the official documentation.' This clearly signals the primary use case, but it lacks explicit mention of alternatives or when not to use the tool, so it doesn't reach a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_country_rulesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden and it does well: 'Return' signals a safe read, and the description discloses exactly what the response contains (currencies, personalId, constraints, notes). It adds the workflow context of calling before payload emission. It doesn't cover failure behavior for unsupported countries, but the core behavior is transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences of densely packed information: the purpose plus content enumeration, followed by a single directive. No filler, and the most important constraint (when to call) is placed clearly at the end.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple lookup with 2 fully-documented parameters and no output schema, the description compensates well by detailing what the return value contains. The only minor omission is error handling for unsupported country codes, but the schema already lists the supported countries, so the description is complete for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so both 'country' and 'method' are already well documented in the schema, including the supported values and formatting rules (uppercase, ISO code). The description adds no parameter detail beyond what the schema provides, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Return') and resource ('country-specific rules') and enumerates the exact content: accepted currencies, personalId formats, payment method constraints, and regulatory notes. This clearly distinguishes it from siblings like search_docs and get_endpoint_spec, which serve different lookup purposes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides an explicit when-to-call directive: 'Call this before emitting any country-specific payload.' This frames the tool as a prerequisite step in the integration flow. It lacks explicit exclusions or named alternatives, but the precondition is clear and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_payloadA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses the return format ({errors, warnings, info} with code, path, severity, hint, suggestion) and warns that 'looks fine' payloads still need validation—setting expectations that the tool may catch issues even when the user thinks they're correct. It does not explicitly state non-behavior like 'does not make API calls' or 'no side effects', but the description implies a read-only validation function. It adds substantial context beyond the schema, justifying a 4.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is very long, though it front-loads the critical usage note ('REQUIRED whenever') and structures the validations as a list. Every sentence adds useful detail, but it could be more concise by grouping related checks. For a tool with high complexity, the length is defensible, but it is not tightly packed—several clauses could be trimmed without loss of meaning.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (6 parameters, nested objects, many validation rules) and the absence of an output schema, the description is remarkably complete. It covers all the inputs' roles, the full set of validations, the output structure, and why skipping is risky. An agent would have everything needed to decide whether and how to call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is stated as 100%, so the baseline is 3. The description reinforces parameter roles (e.g., 'operation' selects the rule set, 'environment' affects notificationUrl checks, 'personalIdType' inference) but adds no new parameter info beyond what the schema already describes. It does contextualize why parameters matter, but not enough to exceed baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific, actionable purpose: validating JSON payloads for a defined set of PayRetailers POST endpoints. It enumerates the exact operations and lists the specific validation checks (CPF/CNPJ checksums, amount format, currency/country match, etc.), making the tool's function unambiguous and distinct from sibling tools like search_docs or get_endpoint_spec.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly states when the tool is REQUIRED ('REQUIRED whenever... even if the payload looks correct') and explains why (generic 4xx errors are hard to interpret). It also directs the user to pipe the returned errors/warnings back into the payload, effectively telling them the expected workflow. No exclusions or alternatives are mentioned, but the tool is clearly the primary validator for these operations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• validate_webhook_handlerA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It clearly states the return structure ({errors, warnings, info} with actionable codes) and the specific failure modes it catches (acknowledging after processing, returning 500 on business errors, missing idempotency, etc.), giving an agent a concrete mental model of what the tool will do. It does not go into output detail beyond that, but for a static-analysis/validation tool this is adequate; a 4 is warranted because it is explicit and actionable without being exhaustive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is longer than typical but every sentence earns its place: the usage directive is front-loaded, the evaluation scope is itemized, and the failure-mode list is concrete and useful. The only slight inefficiency is the duplication of some schema-derived concepts (like the retry risks), but overall it's tightly packed with high-signal guidance rather than filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (13 parameters, 6 enums, nested objects, no output schema), the description is remarkably complete. It states the contract (payload fields), the output format, the key failure modes, and even mentions pairing with a related tool. It could include an example of the 'actionable codes', but the description's coverage of usage, scope, and behavior is sufficient for an agent to decide whether to call it and roughly what to provide.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% — every parameter has a substantive description (e.g., ackStrategy explains the three options and warns about 'conditional-on-outcome', idempotencyStrategy says 'by-eventId' is required). The tool description itself adds only a few high-level hints (like the list of failure modes that map to parameters) but does not go beyond what the schema already explains. Since the schema does the heavy lifting, the baseline 3 is appropriate; the description reinforces but does not add new parameter-specific meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific verb ('Analyses') and a precise resource ('a declarative description of the handler... against the PayRetailers contract') and enumerates the exact inputs (URL, processing mode, ack strategy, etc.) and the output shape ({errors, warnings, info}). It also distinguishes itself from the sibling get_webhook_playbook by naming it explicitly, leaving no ambiguity about which tool fits a design-from-scratch task.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The opening word 'REQUIRED' plus the explicit trigger condition ('whenever the user is designing, describing, or about to code a webhook receiver') makes the when-to-use unmistakable. It also states an alternative ('Pair with get_webhook_playbook when designing from scratch') and even cautions that the tool is needed 'even if the design looks fine', closing off the common excuse of skipping validation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_webhook_playbookA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full disclosure burden. It states the tool is a read-only retrieval operation ('Returns') and details all delivered content: event vocabulary, retry policies, signature verification, etc. It does not claim any side effects or mutations. The emphasis that integrators should not rely on memory signals the tool's authoritative nature. This is sufficient behavioral transparency for a query tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is long but densely packed with essential information, organized as a list of contract aspects and retry policies. The mandatory-use warning is front-loaded, and every sentence contributes value. While verbose, the complexity of webhook contracts justifies the detail; there is no redundant filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  There is no output schema, so the description must explain the return value, and it does so extensively: it lists envelope schema, event categories, HTTP response codes, idempotency, replay protection, signature verification, retry policies, and common mistakes. It also explains parameter usage for scoping results. For a reference tool covering a complex integration surface, this is complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% and the three parameters (category, eventType, includeDelivery) are fully documented in the schema with enums and examples. The description only adds a generic suggestion to filter by category or eventType, but does not introduce new semantic meaning beyond the schema. Per the rubric, baseline 3 is appropriate when the schema handles parameter documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb ('Returns') and resource ('canonical PayRetailers webhook contract') and enumerates the exact contents: envelope schema, event vocabulary, HTTP response contract, idempotency, replay protection, signature verification, retry policies, and common pitfalls. This clearly differentiates it from siblings like search_docs or get_endpoint_spec, which are generic or endpoint-specific rather than webhook-contract-specific.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly mandates when to use the tool: 'REQUIRED before implementing any webhook receiver or reconciliation flow' and 'Do not attempt to design a webhook handler from memory — call this tool first.' It also advises filtering by category or eventType for focused answers, giving clear guidance on how to invoke it effectively. No alternatives are mentioned, but the tool is positioned as the definitive prerequisite, so exclusion guidance is unnecessary.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.