Server Quality Checklist

Tool Scores

• task_statusC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It lists possible statuses but does not disclose whether the operation is read-only, error behavior, auth requirements, or what the response structure looks like.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Single sentence, front-loaded, every word earns its place; no wasted text.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The enumeration of statuses provides useful behavioral context, but missing usage guidance, return format, and error handling leaves gaps. For a single-parameter tool with no output schema and no annotations, this is minimally adequate but not complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The sole parameter task_id is not mentioned in the description beyond the implicit 'task'. Schema description coverage is 0%, and the description does not clarify the format, requiredness, or source of task_id.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the verb 'check' with 'task' as the resource and enumerates six possible statuses, making its purpose clear. It differentiates from siblings like start_task and cancel_task, though it doesn't explicitly name alternatives.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus task_result or list_tasks. The status list implies usage but there are no exclusions or alternative recommendations.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_tasksB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses only the ordering (newest first) but omits important behavioral details such as whether completed tasks are included, the meaning of 'recent' (time window), pagination, or any read-only scoping. With no annotations provided, the description carries full responsibility and only partially covers it.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence with no fluff. It conveys core purpose and ordering efficiently, making it easy for an agent to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple listing tool with one optional parameter and an output schema, the description is lacking contextual completeness. It does not clarify the scope of 'recent tasks' (e.g., user-specific, status, time range), which is critical for correct invocation. The output schema mitigates return-value ambiguity, but the missing behavioral context remains a gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter 'limit' has a default of 20 in the schema, but the description does not explain its effect (e.g., maximum number of tasks returned). Since schema coverage is 0%, the description was expected to compensate but adds no meaning beyond the parameter name.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description 'List recent tasks, newest first' clearly identifies the action (list) and resource (tasks) with a specific ordering. It distinguishes itself from sibling tools that manage individual tasks (start_task, task_status, task_result, cancel_task) rather than providing a list overview.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is used to retrieve an overview of recent tasks, but it does not explicitly state when to use it versus alternatives. No exclusions or prerequisites are mentioned, so the usage guidance is implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cancel_taskA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It explicitly reveals the side effect of 'killing its whole process tree,' which is critical for an agent to understand the destructive scope. It does not detail return values or error cases, but for a single-purpose cancel tool, this is substantial context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence of 10 words, front-loaded with the core action ('Cancel a running task') and followed by the key side effect. Every word earns its place; it is an excellent example of concise specification.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no output schema, no annotations), the description provides the essential behavioral context and side effect. It could mention what happens if the task is not running, but for a straightforward cancel operation, the description is largely complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has one parameter (task_id) with 0% description coverage, and the tool description does not mention the parameter at all. While the name 'task_id' is self-explanatory, the description adds no meaning beyond the schema, failing to compensate for the lack of schema documentation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action ('Cancel') on a specific resource ('a running task'), and adds the detail of 'killing its whole process tree,' which distinguishes it from sibling tools like task_status or start_task. It is unambiguous and directly conveys the tool's function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the tool is for running tasks only, but does not explicitly contrast with alternatives or state when to avoid using it. There is no mention of when to use this versus task_status or task_result, leaving the decision to the agent's inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• task_resultA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden of behavioral disclosure. It reveals that stdout and stderr are merged, that it supports both finished and running tasks (enabling live progress peeking), and that tail_lines limits the output. This is useful behavioral context beyond a simple 'get output' statement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is concise and well-structured: a one-line summary followed by a clear Args section. Every sentence adds value, with no redundant or filler text. It is appropriately sized for the tool's simplicity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no output schema and no annotations, the description covers the essential behaviors: what output is returned, how the output is merged, and the ability to inspect running tasks. It does not specify the exact return format, but the description is sufficiently complete for a straightforward output-fetching tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does so by explaining both parameters: task_id is 'the task to inspect' and can be used on running tasks, and tail_lines specifies 'how many trailing lines of output to return.' This adds meaningful semantics beyond the bare schema properties.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Get a task's captured output (stdout+stderr merged).' The verb 'Get' plus the resource 'task's captured output' is specific, and the scope (stdout+stderr) distinguishes it from sibling tools like task_status or cancel_task.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides clear usage context by noting it works for 'finished AND still-running tasks, so you can peek at live progress.' This implies when to use it (to inspect output, especially for in-progress tasks), though it does not explicitly name alternative tools or state when not to use it.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• start_taskA
  5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully discloses behavioral traits: the command runs in the background, the call returns immediately, the task can be killed if it exceeds timeout_seconds, and a new task id and initial status are returned. This is sufficient transparency for the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with a one-sentence summary, a clear Args block, and a follow-up note. Every sentence adds value, and the format makes it easy to scan and understand without unnecessary verbosity.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple with three parameters and no output schema, yet the description covers what is returned (task id and initial status), how the task continues to run, and how to follow up using sibling tools. This provides complete context for an agent to use the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, but the description provides detailed semantics for all three parameters: command is exemplified, cwd defaults to the server's cwd, and timeout_seconds causes the task to be killed and marked 'timed_out'. This fully compensates for the lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: 'Start a shell command as a background task and return immediately.' It specifies both the verb and resource, and distinguishes itself from siblings by explaining follow-up with task_status/task_result, making it clear this tool initiates the task while others monitor or cancel.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells the agent when to use this tool (to start a background command) and what to do after (use task_status / task_result to follow it). It also provides a specific guideline for timeout_seconds, recommending it for unattended jobs, which helps the agent decide when to set this parameter.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.