Server Quality Checklist

Tool Scores

• comex_trade_flowA
  3.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses significant behavioral details: the 'air' channel is corrected by dispatch point because the dataset's modal field is contaminated and should not be used raw, and the response always includes applicable caveats and file provenance. This is valuable transparency about data quality and output structure. It does not cover all typical behaviors (e.g., read-only nature, pagination, rate limits), but the critical data correction and output caveat information is highly informative.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise, two sentences, and front-loaded with the main purpose. The first sentence states the core function, filters, and grouping; the second adds critical caveats about channel correction and response contents. There is no fluff, and the structure is efficient. It earns a 4 rather than 5 because it could be slightly more structured with bullets, but it is already clear and compact.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity of 8 parameters (all optional) and no output schema, the description covers key aspects: it specifies the metrics returned, the correction for the 'air' channel, and that responses always include caveats and file provenance. This is fairly complete for a data retrieval tool. However, it does not mention typical response format details (e.g., units, pagination, or whether results are aggregated by default), and it omits any prerequisites like authentication. These are minor gaps but not critical.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so all 8 parameters already have explanatory text in the schema. The description text itself does not add new semantics for parameters beyond what the schema provides, even though the channel parameter's complex behavior is fully explained in the schema. According to the rubric, when schema coverage is high, the baseline is 3, and the description adds no additional value here.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool provides Brazil foreign trade flow data (value, weight, and value density) from official Comex Stat microdata, including filters and grouping. It mentions the specific resource and the metrics returned, making the purpose evident. It lacks an explicit verb like 'retrieve' but the noun-based phrasing is still clear. It is distinguishable from siblings, which focus on freight, code resolution, and caveats.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description does not mention when to use this tool versus the sibling tools (comex_data_caveats, comex_freight_cost, comex_resolve_code). It provides no explicit conditions for selecting this tool over alternatives, nor any 'when not to use' guidance. The only implicit hint is that it covers trade flows, but that is not sufficient for guiding tool selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• comex_data_caveatsA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosure. It accurately describes a safe read-only behavior (retrieving caveat metadata) and discloses the internal structure of each result (evidence, correction rule, provenance). For a low-risk retrieval tool there is little behavioral hazard left uncovered.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with purpose and usage, and the essential guidance fits in three sentences. The closing rhetorical question adds rhetorical color but no operational information, making it mildly less economical than it could be.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple tool with one optional enum parameter and no output schema, the description is reasonably complete: it explains what the results contain, when to invoke it, and the overall QA intent. It does not specify the return format, but that is a minor gap for this information-retrieval use case.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%: the single 'id' parameter carries an enum and a clear explanation ('Um caveat especifico. Vazio = todos.'), so the schema already documents the parameter. The description adds no additional parameter-level detail, matching the baseline for high coverage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool exposes known defects of the Comex Stat dataset, each with quantified evidence, a correction rule, and verification provenance. It positions itself against 'a data wrapper' generically, distinguishing its QA purpose from the sibling trade-flow/freight tools, though it never names those siblings explicitly.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit when-to-use guidance ('Use ANTES de publicar qualquer numero derivado desta fonte') and mentions a built-in conferencing checklist. However, it does not state when NOT to use it or name the alternatives (comex_trade_flow, comex_freight_cost) that it should be paired with.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• comex_freight_costA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral disclosure burden. It discloses the data source (VL_FRETE and VL_SEGURO fields), the fact it is import-only, that air is corrected by despatch point, and the output units (US$/kg and % of FOB). It does not describe the response shape or potential limitations like missing data or aggregation behavior, but the key behavioral facts are present. A score of 4 reflects that the description goes beyond a bare statement of function.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each earning its place: the first defines the product and units, the second explains provenance and why it is valuable, the third states constraints (import-only and air correction). It is front-loaded with the core definition and avoids redundant words. No fluff or repetition of schema content. This is concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 5 parameters, no output schema, and no annotations. The description effectively communicates what the tool provides, but it does not describe the return format (e.g., whether it returns a table per channel, time series, or aggregated values). Since there is no output schema, the agent is left guessing at the response structure. While the core semantics are clear, the agent may need additional guidance on what to expect, making it incomplete for a complex data tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% for all 5 parameters, so the baseline is 3. The description does not add parameter-specific semantics beyond the schema; it mentions the channel is always included (which relates to group_by) but does not explain details like how NCM prefix matching works or how to combine years/months. The schema already describes each parameter clearly. Thus the description adds no extra value for parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description names a specific resource (freight and insurance declared to customs by transport channel) and clearly distinguishes it from siblings by emphasizing it is the best systematic measure of realized cost. The verb is implicit ('provides') but the scope and source are explicit. It differentiates itself from comex_trade_flow (which likely covers flows) and comex_data_caveats (which explains caveats). This is a clear, distinct purpose.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives a clear constraint: only imports, with export files lacking freight. It also states the air channel is corrected, which is a usage detail. It implies the tool should be used when realized cost is needed ('melhor medida sistematica de custo realizado'), but it does not explicitly name alternatives or say when not to use it. Thus it provides clear context and a condition, but stops short of explicit when-to-use vs. when-not-to-use guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• comex_resolve_codeA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full responsibility for behavioral disclosure. It mentions bidirectional resolution and the URF classification, which adds specific context. However, it does not disclose return format, error handling, or whether partial matches are supported. For a simple lookup this is adequate but not comprehensive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences. The first states the core function and lists the code types, the second provides the rationale. There is no wasted wording, and the key action is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (3 parameters, no output schema, no annotations), the description is essentially complete. It covers purpose, scope, and rationale. It could mention the return structure (e.g., list of matches) but the bidirectional resolver nature makes that non-critical for an agent to call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds value by explaining what each kind represents (e.g., VIA is modal de transporte, URF includes classification aeroporto/porto/outra) and clarifies that query can be a code or a name. This enriches the parameter semantics beyond the schema alone.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb 'Resolve' with the resource 'codigos oficiais para nomes e vice-versa' and enumerates the exact code types (pais, NCM, VIA, URF) including URF's classification. This clearly distinguishes it from sibling tools like comex_trade_flow or comex_freight_cost, which handle completely different operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context: the dataset is all coded, so the tool should be used whenever codes need translating. While it doesn't explicitly state alternatives or when not to use it, the sibling tools are not viable alternatives for this function, so the lack of exclusion is acceptable. The rationale is enough for an agent to infer when to invoke it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.