Server Quality Checklist

Tool Scores

• call_toolB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description does not disclose any behavioral traits such as return value, side effects, rate limits, or error handling. For a tool that invokes other tools, this lack of transparency is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description consists of two sentences with no redundant or irrelevant information. It is tightly written and front-loads the core purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Despite the simple schema, the tool is a meta-tool that executes others. The description fails to explain the return value (the called tool's output) or address error conditions, prerequisites, or synchronization behavior. This leaves the agent without crucial context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has 100% description coverage on both parameters ('name' and 'arguments'), so the schema already defines their purpose. The description adds no extra meaning beyond 'with the given arguments,' resulting in a baseline score of 3.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('call') and the resource ('tool'), and distinguishes from sibling tools like discover_tools and execute_batch by specifying it executes tools discovered via search_tools. The purpose is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description advises to use this tool after discovering tools via search_tools, providing some context. However, it does not explicitly state when not to use it (e.g., for batch operations) or mention alternative tools like execute_batch. The guidance is minimal.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• discover_toolsA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description must disclose behaviors. It states the tool is a read-only search returning ranked definitions in a specific format, which is adequate. However, it omits any mention of side effects, rate limits, or scope (e.g., whether it searches across all modules). The behavior is minimally described but not fully transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences with no extra words. It front-loads the action and efficiently communicates purpose and return format. Every sentence adds value.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (1 param, no annotations, output schema exists), the description covers the core purpose and output. It could mention that results are from all available tools or that it is a read operation, but it is largely complete for a search tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers 100% of parameters (only 'query' with a description). The description rephrases the schema ('Natural language query') without adding new meaning, such as query format, length limits, or examples. Baseline score of 3 is appropriate since schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool searches for tools using natural language, with a specific verb ('Search') and resource ('tools'). It explains the return format (matching definitions ranked by relevance, like list_tools), which differentiates it from siblings like list_modules and plan_query.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for finding tools by description, but provides no explicit guidance on when to use it versus alternatives like list_tools or call_tool. There are no 'when not to use' or exclusion criteria, leaving the agent to infer context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_modulesA
  3.9/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, but the description discloses the read-only nature implicitly. It does not mention auth requirements, rate limits, or return format, though the tool is simple and likely safe.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, no fluff. The purpose is front-loaded, and keywords at the end aid searchability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the output schema exists, the description need not detail return values. It provides enough context to understand the tool's role, though it could mention the structure of the module list.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, and schema coverage is 100% trivially. The description adds value by stating what the output contains (modules with tool counts and descriptions), which goes beyond the empty schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'List all registered API modules with tool counts and descriptions' and positions it as a precursor to discover_tools, distinguishing its purpose from siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly guides the agent to 'Use this to understand what data sources are available before calling discover_tools for specific queries', providing clear context but no when-not or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• plan_queryA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It transparently states the tool is for planning only and directs to 'execute_batch' for execution. It doesn't cover limitations or error behavior, but the planning nature is well communicated.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with core purpose but includes a lengthy keyword list that adds redundancy. It is mostly concise but could be tightened for efficiency.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the tool's role and relationship to 'execute_batch', but lacks examples, parameter guidance, and constraints. With an output schema present, some gaps are acceptable, but parameter semantics are missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, yet the description adds no meaning for parameters 'query', 'top_k', or 'lang'. It fails to describe input semantics beyond schema defaults and enums, requiring the agent to infer.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool plans multi-step queries across Canadian government data APIs and returns a structured execution plan. It uses specific verbs like 'plan' and 'orchestrate', and is easily distinguishable from siblings like 'call_tool' and 'execute_batch'.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly lists use cases (e.g., multi-API planning, cross-module queries) and advises using 'execute_batch' for execution. This provides clear when-to-use guidance and references an alternative sibling.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_batchA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Discloses key behaviors: uses asyncio.gather for parallel execution, per-step error isolation (one failure doesn't cancel others), and accepts specific input formats. With no annotations, this adequately reveals the execution model.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Efficiently structured: purpose first, then behavior, followed by use cases and keywords. Every sentence adds value without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Covers input format, execution model, error isolation, and use cases. With an output schema present, the return values are implicitly documented. Could add timeout details but overall comprehensive for the tool's complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Despite 0% schema coverage, description adds meaning by explaining the `calls` parameter accepts either a plan_query output or raw list of tool call objects. The `lang` parameter is an enum with default, and its description is not needed beyond schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Execute multiple tool calls in parallel and return aggregated results,' effectively distinguishing it from siblings like call_tool (single call) and plan_query (generates plans without execution).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides explicit use cases like 'running multiple tool calls at once, executing a plan from plan_query,' offering clear guidance on when to use. Could improve by mentioning when not to use, but positive guidance is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.