Server Quality Checklist

Tool Scores

• cancelA
  3.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It reveals a critical non-obvious trait: confirmed=true is only set on an observed terminal/exit, and the cancel request itself does not guarantee a stop. This is genuinely useful runtime semantics. It does not mention output format or whether the call blocks, but the core behavior is well disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two tightly written sentences. The first states the primary action; the second delivers a crucial caveat that prevents misinterpretation. There is no filler, no repetition of the tool name, and the most important semantic distinction is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with no output schema and no annotations, the description covers the essential cancellation semantics well. However, it leaves gaps: it does not define the return shape or clarify whether the reported confirmation is immediate or requires polling. These omissions are notable because there is no output schema for the agent to lean on.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It never explains the task_id parameter beyond implying it identifies what to cancel. There is no mention of how to obtain the ID, what format it must take, or whether it must come from a run call. The agent can only infer the parameter's role from the tool name and general context.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: "Request cancellation and report whether the stop was CONFIRMED by the native runtime." It precisely scopes the tool's behavior and distinguishes it from siblings like run, wait, and inspect. The added clarification that a cancel request alone is not a stop removes ambiguity about what the tool actually accomplishes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives no explicit guidance about when to call this tool versus alternatives such as wait or inspect. It implies cancellation happens after a task is running, but it never says whether this should follow a run call, whether wait is needed to confirm the stop, or under what conditions cancellation is appropriate. The agent is left to infer usage context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• waitA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden, and it discloses non-busy-polling, bounded blocking, the returned snapshot dict with wait_timed_out flag, and the exact terminal status set. It does not mention errors or side effects, but for a wait operation this is solid coverage.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three dense sentences with no redundancy: core purpose, no busy-poll, return/flag behavior, and terminal-status exclusions. The key scoping statement is front-loaded and every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given only two simple parameters and no output schema, the description is complete enough: an agent knows what to pass, what the timeout means, what the response flag indicates, and which statuses count as completion. No critical gap prevents correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description compensates by explaining timeout_sec behavior: if still running, wait_timed_out=true and call again later. It also clarifies that task_id refers to a task awaiting terminal/blocked state. Parameter names and defaults remain in the schema, but the important operational semantics are added.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb and resource: 'bounded blocking wait for a task to reach a terminal or blocked state'. It also says 'does not busy-poll', which separates it from inspect (passive status check), run (initiation), and cancel (mutation), so an agent can tell it apart from siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Gives clear usage context: wait until a terminal status, handle timeout by re-calling later, and do not treat 'submitted'/'running' as completion. It does not explicitly name alternatives or when-not-to-use, but the guidance is strong enough to direct an agent to correct usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• agentsA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the behavioral burden. The verb 'List' clearly signals a read-only operation, and the description discloses the exact output categories and stats. It does not mention staleness or permission requirements, but those are minor for a zero-parameter enumeration tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is lengthy but dense and purposeful: it front-loads the core list action, then presents output fields and routing strategy. Some field enumeration overlaps with the output schema, but the added interpretation and usage guidance justify the length.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a no-parameter listing tool with an output schema and a clear follow-up action (run), this description is complete. It tells the agent what it will receive, how to interpret the stats, and how to apply them when selecting an agent_id. Nothing essential is missing for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and schema description coverage is 100%, so there is no parameter burden for the description to compensate for. The description correctly focuses on output semantics and usage rather than input semantics.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with 'List configured worker agents and their capabilities' — a specific verb and resource — and enumerates the exact stats returned. It also distinguishes itself from lifecycle siblings by explicitly positioning agents as the discovery step before run().

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It says explicitly to use this before run() to pick a valid agent_id, and gives a concrete routing heuristic: high-score agents for hard units, cheaper/lower-score agents for well-specified mechanical units. This is actionable guidance for when and how to use the tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspectA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full behavioral burden. It explicitly discloses that logs/diffs stay on disk, that inspect reads slices rather than whole artifacts, and that `diagnose` returns a sanitized, size-bounded report with redacted secrets/emails/blobs and a stable `code` + `hints`. These are genuinely useful, non-obvious behavioral details.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact, front-loaded with the core purpose, and uses a short list plus one focused bullet for the special `diagnose` behavior. Every sentence earns its place without redundant filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a read tool with no output schema and no annotations, the description covers purpose, allowed `what` values, disk behavior, and the important sanitization guarantee for `diagnose`. It could elaborate on the return shape of the other `what` variants and on offset/limit semantics, but the information provided is sufficient for basic correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema provides zero parameter descriptions and no enums, so the description's explicit `what: status | summary | diff | verify | log | raw | usage | diagnose` list adds real selection value. It does not explain `task_id`, `limit`, or `offset` in detail, but 'read slices here' and the default values make their roles inferable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with a direct verb and resource: 'Read task details on demand,' and then enumerates the concrete read slices via the `what` list. This clearly distinguishes `inspect` from the process-oriented sibling tools (`run`, `wait`, `cancel`, `agents`).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'on demand' signals that inspect is the async/read-side counterpart to running or waiting, and the sibling names make the operational contrast obvious. It does not explicitly name alternatives or exclusion conditions, but the usage context is clear enough for an agent to route correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• runA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden, and it delivers. It discloses timeout behavior (task becomes blocked, never silently resubmitted), budget behavior (task failed with budget_exceeded), idempotency, verify enforcement ('worker cannot pass by editing them'), and the critical nuance that run() success only means SUBMITTED, not done. These are behavioral traits well beyond what any scaffold would imply.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose and is organized as a parameter-by-parameter list, making it scannable. It is long, but each line earns its place given 14 parameters and zero schema descriptions. A small deduction for length, but the structure is effective.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given 14 parameters, 0% schema coverage, no output schema, no annotations, and sibling tools for follow-up, the description is remarkably complete. It covers submission semantics, return meaning, timeout/budget edge cases, and explicitly points to wait()/inspect() for next steps. Nothing an agent needs to call it correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate entirely, and it does. Each of the 14 parameters is explained with purpose and constraints—e.g., session_ref requires same agent+workspace, verify prefers list form, scope_files must be relative. The description adds meaning far beyond the raw schema names and types.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb-resource pair: 'Submit a complete work unit to a worker agent and return its task_id.' This clearly distinguishes the tool from siblings like agents, wait, inspect, and cancel by naming the submission action and the immediate return. The scope is unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit requirements and behavior for each parameter, including constraints like 'complete work unit; do not ping-pong per file/line' and 'absolute or escaping paths are rejected.' It also tells the agent to follow up with wait()/inspect() after a successful submission, which is clear when-to-use guidance versus the siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.