Server Quality Checklist

Tool Scores

• call_toolA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden for behavioral disclosure. It merely restates the schema without describing side effects, return values, error conditions, or the fact that calling arbitrary tools may have destructive consequences. This is a significant transparency gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exactly two sentences, front-loaded with the primary action and followed by a usage hint. Every word earns its place, with no redundancy or fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple, but with no output schema and no annotations, the description could have mentioned that the tool returns the called tool's output or that invalid names will error. The core functionality is clear, but some behavioral expectations are missing, so it is minimally viable rather than complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, with both parameters described. The description adds little beyond the schema, repeating 'by name with the given arguments' without explaining how arguments map to the called tool's parameters. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: 'Call a tool by name with the given arguments,' which specifies the verb and resource. It also differentiates from search_tools by noting the tool is for executing discovered tools, but it does not explicitly distinguish from the sibling executar_lote, so it falls short of a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context: 'Use this to execute tools discovered via search_tools.' This tells the agent when the tool is appropriate, but it does not mention exclusions or alternatives, so it is not a full 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• discover_toolsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behaviors. It states the tool is a read-only search returning ranked definitions in a specific format, which is adequate. However, it omits any mention of side effects, rate limits, or scope (e.g., whether it searches across all modules). The behavior is minimally described but not fully transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no extra words. It front-loads the action and efficiently communicates purpose and return format. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (1 param, no annotations, output schema exists), the description covers the core purpose and output. It could mention that results are from all available tools or that it is a read operation, but it is largely complete for a search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers 100% of parameters (only 'query' with a description). The description rephrases the schema ('Natural language query') without adding new meaning, such as query format, length limits, or examples. Baseline score of 3 is appropriate since schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches for tools using natural language, with a specific verb ('Search') and resource ('tools'). It explains the return format (matching definitions ranked by relevance, like list_tools), which differentiates it from siblings like list_modules and plan_query.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for finding tools by description, but provides no explicit guidance on when to use it versus alternatives like list_tools or call_tool. There are no 'when not to use' or exclusion criteria, leaving the agent to infer context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_modulesA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, but the description discloses the read-only nature implicitly. It does not mention auth requirements, rate limits, or return format, though the tool is simple and likely safe.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, no fluff. The purpose is front-loaded, and keywords at the end aid searchability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, the description need not detail return values. It provides enough context to understand the tool's role, though it could mention the structure of the module list.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, and schema coverage is 100% trivially. The description adds value by stating what the output contains (modules with tool counts and descriptions), which goes beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all registered API modules with tool counts and descriptions' and positions it as a precursor to discover_tools, distinguishing its purpose from siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly guides the agent to 'Use this to understand what data sources are available before calling discover_tools for specific queries', providing clear context but no when-not or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• plan_queryA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It transparently states the tool is for planning only and directs to 'execute_batch' for execution. It doesn't cover limitations or error behavior, but the planning nature is well communicated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with core purpose but includes a lengthy keyword list that adds redundancy. It is mostly concise but could be tightened for efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's role and relationship to 'execute_batch', but lacks examples, parameter guidance, and constraints. With an output schema present, some gaps are acceptable, but parameter semantics are missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet the description adds no meaning for parameters 'query', 'top_k', or 'lang'. It fails to describe input semantics beyond schema defaults and enums, requiring the agent to infer.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool plans multi-step queries across Canadian government data APIs and returns a structured execution plan. It uses specific verbs like 'plan' and 'orchestrate', and is easily distinguishable from siblings like 'call_tool' and 'execute_batch'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly lists use cases (e.g., multi-API planning, cross-module queries) and advises using 'execute_batch' for execution. This provides clear when-to-use guidance and references an alternative sibling.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_batchA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses key behaviors: uses asyncio.gather for parallel execution, per-step error isolation (one failure doesn't cancel others), and accepts specific input formats. With no annotations, this adequately reveals the execution model.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Efficiently structured: purpose first, then behavior, followed by use cases and keywords. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers input format, execution model, error isolation, and use cases. With an output schema present, the return values are implicitly documented. Could add timeout details but overall comprehensive for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema coverage, description adds meaning by explaining the `calls` parameter accepts either a plan_query output or raw list of tool call objects. The `lang` parameter is an enum with default, and its description is not needed beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Execute multiple tool calls in parallel and return aggregated results,' effectively distinguishing it from siblings like call_tool (single call) and plan_query (generates plans without execution).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit use cases like 'running multiple tool calls at once, executing a plan from plan_query,' offering clear guidance on when to use. Could improve by mentioning when not to use, but positive guidance is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.