Server Quality Checklist

Tool Scores

• cancel_commandC
  2.6/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It mentions 'Send a signal' but doesn't clarify what the signal does (e.g., cancels, terminates, pauses) or any side effects (e.g., whether the command stops immediately, if data is lost, or if it requires specific permissions). For a tool that likely mutates command state, this lack of detail is a significant gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence that directly states the tool's action without unnecessary words. It's front-loaded with the core purpose ('Send a signal to a running command'), making it easy to parse quickly. Every word earns its place, adhering to best practices for conciseness in tool definitions.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (interacting with running commands, likely mutative), lack of annotations, 0% schema coverage, and no output schema details in the context, the description is incomplete. It doesn't cover behavioral traits, parameter meanings, or usage context, leaving an agent with insufficient information to invoke the tool correctly or understand its effects, despite the presence of an output schema.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, meaning parameters are undocumented in the schema. The description adds no meaning beyond the schema: it doesn't explain what 'command_id' refers to (e.g., an identifier from 'run_command') or what 'signal_value' represents (e.g., numeric signals like SIGINT, string names, or null for default). With two parameters and no compensation in the description, this leaves key semantics unclear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose3/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states the action ('Send a signal') and target ('to a running command'), which clarifies the tool's basic purpose. However, it's vague about what 'signal' means (e.g., cancellation, interruption, or other signals) and doesn't distinguish it from sibling tools like 'run_command' or 'send_stdin', which might also interact with commands. This leaves room for ambiguity in understanding the exact function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites (e.g., that a command must be running), exclusions, or how it differs from sibling tools like 'send_stdin' (which might send input) or 'manage_file' (unrelated). Without such context, an agent might struggle to select this tool appropriately in scenarios involving command management.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• manage_fileC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. While it mentions operations (create, edit, inspect), it doesn't address critical behavioral aspects: whether these operations are destructive, what permissions are required, how errors are handled, or what the output looks like. For a tool with 21 parameters and file system operations, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise - a single sentence that efficiently communicates the core functionality. Every word earns its place: 'Create, edit, and inspect' covers the main operations, 'files on the remote host' specifies the resource and context, and 'including line-based operations' adds important scope information without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity (21 parameters, file system operations), no annotations, and 0% schema description coverage, the description is insufficient. While an output schema exists, the description doesn't address critical context: mutation implications, error conditions, permission requirements, or how the numerous parameters interact. For such a complex tool, the single-sentence description leaves too many questions unanswered.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  With 0% schema description coverage and 21 parameters, the description provides minimal parameter context. It mentions 'line-based operations' which hints at parameters like 'line', 'start_line', and 'end_line', but doesn't explain how these parameters interact or what other parameters like 'escape_profile' or 'anchor' mean. The description doesn't compensate for the complete lack of schema descriptions.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Create, edit, and inspect files on the remote host including line-based operations.' It specifies the verb (create/edit/inspect), resource (files), and scope (remote host, line-based operations). However, it doesn't explicitly differentiate from sibling tools like run_command or send_stdin, which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives. It doesn't mention prerequisites, constraints, or compare it to sibling tools like run_command for file operations. The phrase 'including line-based operations' hints at capabilities but doesn't establish clear usage boundaries.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_commandC
  2.8/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries full burden. It mentions 'live stdout/stderr streaming' which adds behavioral context beyond the schema, but fails to disclose critical traits like security implications, permission requirements, potential destructive effects, rate limits, or error handling. For a shell execution tool, this is a significant gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Perfectly concise single sentence that front-loads the core purpose. Every word earns its place with no wasted text or redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given high complexity (shell execution with 6 parameters), no annotations, and 0% schema coverage, the description is inadequate. While an output schema exists, the description doesn't address security concerns, error cases, or parameter usage that are critical for this type of tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It mentions no parameters at all, leaving all 6 parameters (command, working_directory, environment, timeout, command_id, allocate_pty) undocumented. The description adds no meaning beyond what the bare schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('execute') and resource ('shell command') with the specific behavior of 'live stdout/stderr streaming'. It distinguishes from sibling tools like cancel_command and send_stdin by focusing on execution rather than management or input. However, it doesn't explicitly differentiate from manage_file which might also involve execution.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No explicit guidance on when to use this tool versus alternatives like cancel_command or send_stdin. The description implies usage for executing shell commands with streaming output, but lacks context on prerequisites, when not to use it, or comparisons to siblings.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• send_stdinC
  2.9/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It states the action but doesn't describe what happens (e.g., whether input is buffered, if it blocks, error conditions like invalid command_id, or side effects). This leaves critical behavioral traits unspecified for a tool that interacts with running processes.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, efficient sentence with zero wasted words. It's front-loaded with the core purpose and appropriately sized for the tool's complexity, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity (interacting with running commands) and lack of annotations, the description is minimally adequate but incomplete. The presence of an output schema helps, but the description doesn't address behavioral nuances or parameter meanings, leaving gaps in understanding how to use it effectively.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate but adds no parameter information. It doesn't explain what 'command_id' refers to, what 'data' should contain, or the meaning of 'eof' (likely 'end-of-file'). This leaves all three parameters semantically unclear beyond their schema types.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Forward input') and target ('to a running command's stdin pipe'), providing a specific verb+resource combination. However, it doesn't explicitly differentiate from sibling tools like 'run_command' or 'cancel_command', which prevents a perfect score.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives like 'run_command' or 'cancel_command'. It doesn't mention prerequisites (e.g., needing a running command), exclusions, or contextual triggers, leaving the agent to infer usage from the purpose alone.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.