Server Quality Checklist

Tool Scores

• task_adoptC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations exist, so the description must disclose behavior. It lists three internal checks, which is useful, but it omits side effects, outcome, or return value—essentially whether the tool actually adopts the task or merely tests adoptability.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness4/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The entire description is one concise, front-loaded sentence with a clear list structure. The telegraphic style and jargon reduce readability slightly, but no words are wasted.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a specialized tool with a non-obvious adoption workflow and no output schema or annotations. The description omits parameter semantics, return format, side effects, and usage context, making it insufficient for confident invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter jobDir is neither described nor contextualized in the description. The mention of lock/pid/startSec hints that jobDir contains a lock file, but the link is implicit and the description adds no concrete meaning for the parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description specifies an adoption check with three concrete evidence components (lock pid:startSec, process liveness, exit protocol read), giving a clear function distinct from sibling run/wait/output/autopsy/kill tools. However, it never explains what 'adoption' means or what the tool actually does beyond checking.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'adoption check' implies the tool is for adopting a task, but there is no explicit statement of when to use it over alternative tools like task_run, task_wait, or task_autopsy. No exclusions or alternatives are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• task_waitB
  3.4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description reveals the terminal-state detection mechanism ('EXIT:<code> written'), which is useful behavioral context beyond the schema. However, it does not disclose timeout behavior (despite a timeoutMs parameter), what happens on timeout, or whether the wait is polling-based.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single concise sentence that immediately conveys the core purpose. No unnecessary words or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is adequate for a simple wait operation but leaves gaps: no return value explanation, no timeout/error behavior, and no mention of prerequisites. Given the lack of annotations and output schema, the description should provide more context but is not completely inadequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters1/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The description provides no information about jobDir or timeoutMs beyond their names. With 0% schema description coverage, the description fails to compensate, leaving parameter semantics entirely to the agent's inference.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: waiting for a task to reach a terminal state. The phrase 'EXIT:<code> written' gives a specific completion condition, and the verb 'wait' distinguishes it from siblings like task_run or task_kill.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage after launching a task (e.g., task_run), but it does not explicitly state when to use this vs. alternatives or mention any prerequisites. The context of sibling tools suggests the typical use case, but there is no direct guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• task_autopsyA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of disclosure. It reveals the output format (manner/evidence/verdict/death-code D-01~D-09), providing useful behavioral insight into what the report contains. However, it does not state whether the operation is read-only, what side effects occur, or any prerequisites like task completion status.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that efficiently communicates the tool's purpose and output format without unnecessary words. It front-loads the key action and includes specific format details in parentheses, making it highly concise and well-structured.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (one parameter, no annotations, no output schema), the description covers the essential purpose and the report format, which compensates for the lack of an output schema. It is largely complete for a basic report-generating tool, though it could add usage guidance and parameter edge-case details for full completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  There is one required parameter, jobDir, and the schema provides no description (0% coverage). The description clarifies that jobDir is a 'task directory', adding minimal meaning beyond the parameter name. It does not specify path format, existence requirements, or whether relative/absolute paths are accepted, so it only partially compensates for the missing schema details.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool generates an autopsy report for a task directory, using the specific verb 'generate' and a distinct resource. It also specifies the format (autopsy-spec with manner/evidence/verdict/death-code), which distinguishes it from sibling tools like task_run or task_kill.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage for diagnosing a task's outcome, but it does not explicitly state when to choose this tool over siblings or provide exclusions. The context is somewhat implied by the tool's name and purpose, but no explicit guidance or alternatives are mentioned.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• task_outputA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description bears the full burden of disclosing behavior. It mentions the incremental read capability via byte offset, but it does not explicitly confirm that the operation is non-mutating, nor does it describe any prerequisites, error conditions, or side effects. The verb 'read' implies safety but does not guarantee transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the primary purpose and includes the key modifier about incremental reads. Every word adds value; there is no redundancy or padding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read tool, the description provides sufficient context: it names the file, implies the directory, and explains the offset parameter. With no output schema and no annotations, it could elaborate on the return format or error behavior, but for standard use cases this is adequate.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides zero descriptions for the two parameters. The description adds meaning by linking 'out.log' to the first parameter (likely jobDir is the directory containing that log) and explaining 'offset' as a byte offset for incremental reads. However, it does not explicitly define jobDir, leaving some ambiguity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Read task output') and the specific resource ('out.log'), with an additional detail about byte-offset support. This distinguishes it from sibling tools like task_run or task_kill, which handle lifecycle operations rather than reading output.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage when needing to read a task's output, but it does not explicitly state when to use this tool versus alternatives like task_wait or task_autopsy, nor does it mention any exclusions or prerequisites. The context is inferred rather than articulated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• task_runA
  3.9/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses key behaviors: the sandbox environment (ACL/bwrap/sandbox-exec), the return value (job directory), and side effects (writes lock=pid:startSec and EXIT:<code>). This goes beyond a simple 'runs a command' but does not explain timing or error handling fully.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, well-structured sentence that front-loads the primary purpose and packs in return and side-effect information without wasted words. Every phrase earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 2-parameter tool with no output schema, the description covers the core action, environment, return value, and completion signaling. It could mention integration with sibling tools (e.g., use task_wait after) or clarify async behavior, but it is largely complete for its complexity.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is only 50% (the required 'command' is described, but optional 'label' is not). The description adds no parameter-level detail and does not compensate for the missing label semantics. It only reinforces that 'command' is executed, which the schema already states.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Run' and identifies the resource ('command inside the evidence-protected sandbox runner'). It clearly distinguishes from sibling tools by focusing on execution, while siblings handle waiting, output, kill, etc.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies this tool is for starting a command, but it does not explicitly state when to use it versus alternatives like task_wait or task_output. No explicit exclusions or 'when not to use' guidance is provided, so it relies on context from sibling names.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• task_killA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the SIGKILL signal, indicating a forceful, uncatchable kill, and notes its experimental purpose. It does not detail side effects or error behavior, but the SIGKILL mention provides meaningful behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One sentence, front-loaded with the action verb, and every clause adds value. It is compact and readable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter destructive tool, the description covers what is killed, how, and why. It lacks details about return values or handling of non-running tasks, but given the intentional simplicity and lack of output schema, it is reasonably complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only has jobDir with no description, and the description adds that the kill is 'by its lock pid', implying jobDir is the task directory containing that pid. This linkage is helpful but indirect; it could be more explicit about the relationship between jobDir and the lock pid.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb ('Kill'), identifies the resource ('running task'), and specifies the mechanism ('SIGKILL' via lock pid), clearly distinguishing it from sibling tools like task_wait or task_adopt.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The phrase 'for crash/adoption experiments' gives clear context for when to use the tool, implying it's for fault injection rather than normal task termination. However, it does not explicitly mention when not to use it or name alternative tools.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.