Server Quality Checklist

Tool Scores

• call_toolC
  2.1/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, and the description fails to disclose any behavioral traits such as side effects, error handling, or idempotency. The description says nothing beyond the basic action, leaving the agent with no insight into behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely short (one sentence), but this is due to under-specification rather than conciseness. Essential details are missing, making it insufficient for effective tool selection and use.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  With no output schema, no annotations, and a generic purpose, the description is completely inadequate. It does not explain what returns, error behavior, or the concept of 'data tool'. The complexity of the tool (which may call other tools) demands more context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, and the description adds no meaning beyond the parameter names. It does not explain what 'arguments' should contain, what types are expected, or any constraints. The description provides no added value for parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the verb 'invoke' and specifies it operates on a 'data tool' by name with arguments. This is clear in stating the action and resource, but lacks differentiation from sibling tools like execute_batch which might also invoke tools. Without context, the distinction is unclear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like execute_batch or discover_tools. There is no mention of prerequisites, limitations, or typical use cases.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• discover_toolsB
  3.2/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It states that the tool returns ranked tools with their schemas, which is helpful. However, it does not disclose any other behavioral traits, such as whether the operation is read-only (likely discoverable), has side effects, or requires specific permissions. For a discovery tool, these details are important for an agent to safely invoke it.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise: two sentences that convey the core purpose and a usage hint. It is front-loaded with the primary action ('Find data tools...') and then adds supporting detail. There is no unnecessary text. However, it could be more structured (e.g., bullet points for key aspects), but given its brevity, it is effective.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has three parameters (with no schema descriptions), no annotations, and an output schema that likely explains the return format, the description should cover parameter semantics to be complete. It does not. The intended use as a discovery tool is stated, but agents need to understand the 'module' and 'limit' parameters to use it correctly. Thus, the description is incomplete for its complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has three parameters ('query', 'module', 'limit') with no descriptions (schema coverage 0%), and the description does not explain any of them. The description only says 'Find data tools relevant to a natural-language need,' which vaguely relates to 'query' but does not clarify 'module' or 'limit.' Without parameter explanations, the description adds no semantic value beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Find data tools relevant to a natural-language need.' It also specifies the output: 'Returns ranked tools with their argument schemas.' Additionally, it distinguishes itself from the sibling tool 'call_tool' by suggesting a usage order: 'Use this first, then invoke the chosen tool via call_tool.' This provides strong differentiation from its primary sibling.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides a clear usage guideline: 'Use this first, then invoke the chosen tool via call_tool.' This indicates a typical workflow. However, it does not explicitly mention when not to use this tool or compare it to other siblings like 'list_modules' or 'plan_query,' which could help agents decide between them. The guidance is good but not exhaustive.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• plan_queryA
  3.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description must bear full responsibility for behavioral disclosure. It only states it 'surface[s] candidate tools' but omits details on side effects, auth requirements, or output format (though output schema may exist).

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, immediate verb-first statement, and no filler. Every sentence adds context.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool has one parameter and an output schema, the description provides purpose and usage context but lacks detail on the plan structure and potential constraints. It is adequate but not thorough.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description adds little value beyond the parameter name 'goal'. It merely says 'for a broad goal', which is a restatement.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool 'surface[s] candidate tools across modules for a broad goal', using a specific verb and resource, and distinguishes from siblings by focusing on multi-step planning.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly targets 'multi-step needs' and provides an example (search dataset then query datastore), but does not mention when not to use it or list direct alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_batchA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations provided; the description covers concurrency and error isolation but lacks details on ordering, batch size limits, or argument validation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise (two sentences plus a code snippet), front-loaded with the main action, and each sentence serves a purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the presence of an output schema and only one parameter, the description is fairly complete, though it could mention batch size limits or execution order.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage, the description adds meaning by specifying the structure of the 'calls' parameter as a list of objects with 'name' and 'arguments', beyond the schema's generic object type.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states 'Run several tool calls concurrently', which specifies the verb 'run' and resource 'tool calls', distinguishing it from siblings like call_tool (single call) and discover_tools.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies when to use it (batch multiple calls with error isolation) but does not explicitly mention when not to use or alternatives like call_tool for single calls.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_modulesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The behavior is straightforward: list modules with counts. No annotations are provided, but the description fully covers what the tool does without hidden side effects or mutations.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence with the verb front-loaded, no redundant information, and perfectly sized for the tool's simplicity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given zero parameters and an output schema that likely details the module list, the description sufficiently explains what the tool returns. No additional context is needed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There are no parameters, and the schema covers 100% of input specification. The description adds no parameter information, which is acceptable as there are none to document.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists data-source modules and their tool counts, using a specific verb+resource. It distinguishes from siblings which perform different actions like calling tools or planning queries.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for getting an overview of available modules, but provides no explicit guidance on when to use this tool versus alternatives. No exclusion criteria or usage context is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.