Server Quality Checklist

Tool Scores

• refreshA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already indicate it's a non-read, idempotent operation. The description adds the cache TTL bypass and the 'updates all' default behavior, providing useful context beyond the annotations without contradicting them.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences, front-loaded with the primary action. Every word adds value, and it avoids redundancy with the schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with one optional parameter and no output schema, the description covers the purpose, usage, and key behavioral aspects (TTL, default scope). It could mention outcome details, but given the simplicity, it's sufficiently complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema description already explains the 'domain' parameter with 100% coverage, and the tool description reinforces it with the same idea. The description adds little beyond naming the parameter as a domain id and noting that omission updates all, which is already in the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool forces re-extraction of public sources, bypassing the 72h cache TTL. This is a specific action distinct from sibling tools like get_item or search, and explicitly mentions its scope.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear when-to-use guidance ('Use quando suspeitar de dados desatualizados') and explains behavior with and without the domain parameter. It doesn't explicitly contrast with alternatives, but that's not essential for a refresh operation.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_itemA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description reveals behavioral details beyond annotations: for *-openapi and participantes domains, the full spec node is included in `detail`; $ref fields are not expanded but listed in `refs`; and pcm-additional-info returns the full record vs search's summary. These details are not present in the annotations, which only cover reading/idempotency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the primary purpose, then provides essential domain-specific nuances. No wasted words; every clause adds necessary information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of multiple domains and no output schema, the description covers all key variations: full vs summary records, the `detail` field, and the `refs` behavior with instructions to resolve. It is complete enough for an agent to know what to expect and how to proceed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already has 100% coverage with descriptions for both `id` and `domain`. The description adds contextual meaning by specifying that `id` comes from search and that domain-specific behavior affects the response, which is valuable beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Devolve o registro completo de um item pelo id retornado por search', which is a specific verb+resource. It also distinguishes from siblings by noting that search returns only a summary in pcm-additional-info, while get_item returns the full record.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly instructs to use get_item with an id from search and to call get_item on referenced refs in *-openapi domains to resolve them. This provides clear when-to-use guidance and contrasts with search, which returns summaries in some domains.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_domainsA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Annotations already declare readOnly, idempotent, and non-destructive. The description adds meaningful context about the returned content (domains with filters and local cache state), going beyond the annotations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  A single, front-loaded sentence that conveys purpose, content, and usage flow with no redundant words. Every clause adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no parameters, the description sufficiently explains what the tool returns (domains, filters, cache state) and how it fits into a larger workflow.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the baseline is 4. The schema is empty and the description correctly does not spend space on parameter details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists knowledge domains for Open Finance Brasil, including accepted filters and cache state. It explicitly distinguishes it from siblings by saying 'Comece por aqui' and pointing to search and get_item as next steps.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'Start here; then use search(...) and get_item(...)', giving a clear when-to-use directive and naming alternatives. This is strong usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• searchA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Beyond the annotations (readOnlyHint, idempotentHint, openWorldHint), the description discloses important behavioral traits: compact return (omits nulls), first-query domain extraction delay (~30s), and the semantic of `filters` and `query`. This adds significant context about performance and response structure that annotations alone do not convey.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured. Each sentence adds value: purpose, filter/query semantics, return format, linkage to get_item, and performance warning. No fluff or unnecessary detail, while remaining highly informative.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (5 parameters, nested filters, no output schema), the description is remarkably complete. It covers return format, use of sibling tools, domain extraction delay, and filter/query behavior. No critical operational aspect is missing for an agent to invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so baseline is 3. The description adds meaningful semantics by explaining that `filters` accepts keys from `list_domains` (AND-combined) and that `query` does substring matching on text fields. This goes beyond the schema's syntax, though other parameters like `limit` and `offset` are already self-explanatory.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Busca filtrada em um domínio' (filtered search in a domain), specifying the verb and resource. It distinguishes from sibling tools by emphasizing search over retrieval (get_item), listing (list_domains), and refresh actions, making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit guidance on when to use the tool and how it relates to alternatives: `filters` keys come from `list_domains`, and each result has an `id` to use with `get_item`. This effectively tells the agent when to choose search versus retrieval tools, though it doesn't state a 'when not to use' scenario explicitly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.