Server Quality Checklist

Tool Scores

• read_fileB
  3.3/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries full responsibility for behavioral disclosure, but it only states the basic read action. It does not mention error handling, file size limits, encoding, or the safe/read-only nature of the operation, which is a notable gap.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that conveys the essential purpose without any fluff. Every word is meaningful, and it is immediately readable.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple read tool, the description is adequate but not complete. It does not describe what the tool returns (file content) or potential error scenarios. Given no output schema and no annotations, the description could have provided more context about the behavior and output.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema covers both parameters with clear descriptions ('Sandbox ID' and 'Absolute file path to read'), so baseline is 3. The description adds the qualifier 'text file', which is not in the schema, but this is minor and does not significantly enhance parameter understanding.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('Read') and resource ('a text file from a sandbox'), distinguishing it from sibling tools like write_file and list_files. The verb and object are specific and unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides no guidance on when to use this tool versus alternatives (e.g., run_command could also read files). There are no stated exclusions or preferred contexts, leaving the agent to infer usage solely from the tool name.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_sandboxB
  3.4/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description must disclose behavioral traits itself, but it only says 'secure' and 'returns a sandbox_id'. It does not explain what 'secure' entails (e.g., isolation, network restrictions), whether creation is side-effecting, resource costs, or behavior on timeout/cleanup. This is a significant gap for a create operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, tightly written, and front-loaded with the core action and output. Every word earns its place, with no redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness2/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is complex (5 optional parameters, security presets, templates, network allowlists) and has no output schema, but the description only mentions returning a sandbox_id. It does not explain default behavior, how security presets affect the environment, or how to choose between templates. Given the complexity and lack of annotations, the description is too sparse to guide correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema provides 100% description coverage for all 5 parameters, including details about defaults, enums, and constraints. The tool description itself adds no parameter-specific information beyond the schema, so a baseline score of 3 is appropriate because the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb 'Create' and the resource 'secure sandbox environment', and distinguishes this from siblings like kill_sandbox and list_sandboxes by focusing on creation. It also mentions the key output (sandbox_id) for use with other tools, making the tool's role unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies the sandbox is a prerequisite for other sandbox tools ('to use with other tools'), giving some context for when to call it. However, it does not explicitly state when to use this tool versus alternatives (e.g., running commands directly without a sandbox) or provide exclusions, so the guidance is implied rather than explicit.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_commandA
  3.5/5.0
  Behavior2/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of disclosing behavior. It only states the return values (stdout, stderr, exit code) but fails to mention that commands can have side effects within the sandbox, potential resource implications, or behavior on timeout. This is a significant gap for a command execution tool that could be destructive.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two short sentences, front-loaded with the core action and immediately followed by return value information. Every word earns its place; no fluff or repetition of schema details.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the basic action and outputs but lacks context on prerequisites (e.g., an active sandbox), potential side effects, or error handling. Given the tool's complexity (arbitrary command execution) and lack of output schema/annotations, it is adequate but incomplete for an agent to fully understand the tool's safety and operational context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already fully documents all three parameters. The description adds no extra semantic meaning beyond what is in the schema; it doesn't explain nuances like the timeout range or how the sandbox_id is obtained. Baseline 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states a specific action ('Execute a shell command') and its context ('inside a sandbox'), distinguishing it from sibling tools like create_sandbox and read_file. The verb + resource combination is precise and unambiguously identifies the tool's function.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage by specifying 'inside a sandbox,' but does not explicitly state when to use this tool versus alternatives or provide exclusions. It also doesn't mention that the sandbox must already exist (though the schema hints at this via the sandbox_id description). No direct comparison with sibling tools is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• kill_sandboxA
  3.6/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  The description discloses the core destructive behavior and that resources are released, which is essential. However, with no annotations provided, it does not mention irreversibility, potential failure modes, or effects on running processes. The word 'Destroy' implies finality, but richer context would strengthen this.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  One concise sentence that is front-loaded and contains no filler. Every word contributes to understanding the tool's purpose.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter destruction tool with no output schema, the description covers the essential action and outcome. It could mention error handling or whether the operation is idempotent, but the simplicity of the tool allows for adequate completeness.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already documents sandbox_id with a clear description, and coverage is 100%. The tool description adds no additional parameter meaning, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb 'Destroy' with a clear resource 'sandbox', immediately distinguishing it from sibling tools like create_sandbox and list_sandboxes. The phrase 'release its resources' adds additional clarity about what destruction entails.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines2/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  No guidance is given on when this tool should be used versus alternatives, nor are any prerequisites or exclusions mentioned. The description simply states what it does, leaving the agent to infer that it is for decommissioning sandboxes.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_filesA
  3.7/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It states the basic read-only behavior (listing), which is safe and non-destructive. However, it does not disclose details such as whether the listing is recursive, includes hidden files, or how errors are handled (e.g., if path does not exist). This is acceptable for a simple listing tool but lacks depth.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single sentence that directly states the tool's purpose without any filler. It is front-loaded with the verb 'List' and includes necessary context (sandbox). Every word contributes, achieving high conciseness.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 2-parameter tool with no output schema, the description is adequate but not complete. It does not explain the return format, whether listing is recursive, or behavior on invalid paths. Given the lack of output schema, more details about output and edge cases would improve completeness, though the tool is simple enough that the current description is minimally viable.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema covers both parameters with descriptions (sandbox_id: 'Sandbox ID', path: 'Directory path to list (default: /)'), yielding 100% schema coverage. The description adds no additional parameter-level semantics beyond what the schema already provides, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool lists files and directories at a given path in a sandbox, providing a specific verb ('list') and resource ('files and directories at a given path in a sandbox'). It distinguishes from sibling tools like read_file and write_file by focusing on listing directory contents rather than file content operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: when you need to list directory contents in a sandbox. However, it does not explicitly mention when not to use it or contrast with alternatives such as read_file or run_command. The context is clear enough but lacks explicit exclusions or alternative references.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_sandboxesA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It correctly indicates a non-destructive list operation and adds context about 'active' status and 'current API key'. However, it does not disclose details like pagination, sort order, or the structure of the returned objects, which could matter for a complete behavioral picture.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is a single, front-loaded sentence that states the verb, resource, and scope. Every word earns its place, with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple list operation with no parameters and no output schema, the description adequately conveys the core purpose and scope. It could mention return value structure, but given the simplicity and the sibling tool set, it is reasonably complete. A minor gap is not specifying what fields will be returned for each sandbox.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, and the input schema is empty. According to the rubric, a zero-parameter tool deserves a baseline of 4. The description adds no parameter information, but none is needed since there are no parameters.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'List' with a clear resource ('sandboxes') and scope ('all active sandboxes for the current API key'). This clearly distinguishes it from sibling tools like create_sandbox, kill_sandbox, and file/list operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies its use for retrieving a user's active sandboxes, and the context of 'current API key' adds a filtering condition. However, it does not explicitly mention when to use it instead of alternatives, nor does it state any exclusions or prerequisites.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• write_fileA
  4.1/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure. It does mention the useful behavior of automatically creating parent directories, but it omits important aspects such as overwrite behavior, error handling, permissions, and side effects. For a mutation tool, this is only partially transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two concise sentences with no wasted words. It front-loads the primary purpose and adds a key behavior in the second sentence, making it easy to parse quickly.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple file write operation with three fully described parameters, the description is adequate but has gaps. It lacks information about return values, error conditions, or what happens if the file already exists (overwrite vs. append). This is a significant behavioral detail that should be disclosed.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already covers all three parameters with descriptions, providing a baseline of 3. The tool description adds value by clarifying that content is 'text' (implying not binary) and that parent directories will be created automatically for the path, which goes slightly beyond the schema's description.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb ('Write') and resource ('text content to a file in a sandbox'), and distinguishes from sibling tools like read_file, list_files, and run_command. It also adds a specific behavioral feature (auto-creating parent directories), making the purpose unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description makes it clear this tool is used to write files within a sandbox, which is distinct from reading files (read_file) or listing files (list_files). It does not explicitly mention when not to use it, but the context is sufficient and there are no exclusions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.