Server Quality Checklist

Tool Scores

• process_orderD
  1.5/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, and the description discloses no behavioral traits. It does not mention side effects (e.g., order creation), authorization needs, rate limits, or error behavior, despite the description claiming 'error handling' exists.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness2/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that is not concise because it wastes space on meta-description rather than tool purpose. It is under-specified and does not earn its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the complexity (nested objects, enums, no output schema), the description is completely inadequate. It fails to explain what the tool returns, how errors are handled, or any behavior beyond the raw parameters.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description adds no meaning beyond the input schema. While 3 of 4 parameters have schema descriptions, the missing 'customer' parameter description is not compensated. The meta-commentary does not clarify parameter usage or constraints.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose1/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description is a meta-commentary about the input schema, not the tool's actual purpose. It says 'Demonstrates a tool with a rich input schema' without indicating what the tool does (e.g., creates an order). This is misleading and fails to convey the core function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool vs alternatives like 'fetch_data' or 'manage_file'. The description lacks any context about typical use cases or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• fetch_dataD
  1.6/5.0
  Behavior1/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses no behavioral traits. As a placeholder, it doesn't mention side effects, authentication needs, or other important behaviors. No annotations exist to compensate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness3/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is short but lacks substance. It is concise only because it is a placeholder, not because it efficiently conveys useful information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness1/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is insufficient for real use. It provides no details about return values, error handling, or any context beyond being a template. The tool is not operational as defined.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Although schema coverage is 100%, the description adds no meaning beyond the schema's parameter descriptions. It doesn't explain how parameters affect the API call.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose2/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description is vague, stating it's a placeholder for an API call. It doesn't specify what resource or action it performs, nor does it distinguish from sibling tools like manage_file or process_order.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines1/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance on when to use this tool versus alternatives. The description fails to provide any context about appropriate usage scenarios.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• long_running_taskC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full burden. It only mentions 'async' and 'simulates', but does not disclose idempotency, result format, error behavior, or side effects, leaving critical gaps for an agent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single efficient sentence with no redundancy. It front-loads the key concept of being async and long-running. However, it could be slightly more structured to include usage hints.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema, no annotations, and the need for an agent to understand invocation context, the description fails to explain return values, errors, or expected behavior, making it incomplete for reliable tool selection.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% with 'taskName' and 'durationMs' documented. The description adds no additional meaning beyond the schema, so baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states it simulates a long-running operation with examples (data processing, report generation), but does not clearly differentiate from siblings like 'process_order' or 'fetch_data', which may also be long-running or async.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is provided on when to use this tool versus alternatives like 'fetch_data', 'process_order', or 'manage_file'. The agent is left to infer its purpose without comparative context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• hello_worldA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries full burden. It mentions returning a personalized greeting but does not specify the format or any side effects. For such a simple tool, this is minimally adequate.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the purpose, no unnecessary words, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no annotations, no output schema), the description is mostly complete but lacks detail on the return format (e.g., 'Hello, {name}!'). This is a minor gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% with one parameter described. The description merely restates that it takes a name, adding no new meaning beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states it is a greeting tool that takes a name and returns a personalized greeting, which is a specific verb+resource pattern and distinguishes it from siblings like fetch_data or process_order.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use or not use this tool versus alternatives. It is implied by the simple greeting context, but the description lacks any conditional or comparative language.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• manage_fileA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Without annotations, the description carries full behavioral disclosure. It mentions the base directory constraint but omits important details like error handling, overwrite behavior, or file creation permissions.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise sentences with no redundancy. Critical information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a simple tool with 3 fully documented parameters and no output schema, the description covers the core function and key constraint. Minor gap in error behavior.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the description adds value by clarifying relative paths via 'base directory' but does not exceed the schema's explanations.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool reads and writes local files and specifies a key constraint (configurable base directory). It distinguishes from sibling tools like fetch_data or process_order by focusing on file operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies use for file read/write but provides no explicit guidance on when to use this tool versus siblings, nor any conditions for avoidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.