Server Quality Checklist

Tool Scores

• list_sandbox_dirA
  3.8/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden. It discloses that the tool lists directory contents and returns name/kind/size, which is useful. However, it does not explain behavior for missing paths, permission issues, or the effect of the network parameter — though this is a simple read operation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three concise sentences: purpose, when to use, and return structure. It is front-loaded and every sentence earns its place with no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness3/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is adequate for a basic directory-listing tool and the output schema covers return values. The significant gap is the network parameter, which is neither described in the schema nor in the description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters2/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate for parameter meaning. It does not explain 'path' beyond the obvious default, and 'network' is completely unaddressed — an agent cannot tell from the description what network=true changes.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The first sentence names a specific verb and resource: 'List the contents of a directory inside the sandbox.' The return shape ('name', 'kind', 'size') and the contrast with read_sandbox_file make it easy to distinguish from sibling file operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear when-to-use guidance: use it to see what is actually there before reading or running something, especially after install_sandbox_package or after a read_sandbox_file not_found. It does not explicitly state when not to use it or name alternative tools, so it falls just short of 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• install_sandbox_packageA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden and does well: it discloses that installation persists for the sandbox lifespan and survives until reset_sandbox, and that the returned commands are exactly what run_command can call. It does not mention network behavior or failure modes, but the key side effect is clear.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: the action and outcome come first, followed by when to use it, package format, and persistence. Every sentence adds actionable information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The output schema exists and the description covers the return semantics and persistence, so the agent knows what to expect. The main gap is the undocumented `network` parameter, which could affect whether an install can reach a registry.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The `package` parameter is meaningfully documented with registry-name examples and a version format. However, the `network` boolean is left unexplained in both the schema and the description, so with 0% schema coverage the description only partially compensates.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('Install'), a resource ('Wasmer registry package into the sandbox'), and an outcome ('report the commands it adds'). It also clarifies the tool fills a missing capability, which distinguishes it from run_python and run_command.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Gives an explicit use condition: use it when the sandbox does not already have the tool, since the guest starts with only Python. It also connects the returned commands to run_command, but it does not explicitly list exclusions or alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_sandbox_fileA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  Since no annotations are provided, the description carries the full burden. It discloses important behaviors: long files are capped as head + tail with truncation indicated via `truncated: true`, and the `network` flag selects a sandbox whose files are invisible to the other. This is substantive, though it does not describe error behavior for missing files.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the primary action, then follows with usage context, truncation behavior, and the sandbox-selection caveat. Every sentence contributes information and nothing is redundant.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers the core usage scenario, truncation result shape, and network isolation. The output schema likely covers return fields, so the only notable gap is path handling and missing-file behavior, which are minor for a read tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema has no descriptions and the description adds meaning for the `network` parameter, explaining that it selects which sandbox and that writes to one are not visible in the other. The `path` parameter is left entirely to inference; its format and sandbox-relative semantics are not explained.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: "Read a text file from inside the sandbox." It also names the sibling it is not by referencing `write_sandbox_file`, so an agent can distinguish this read tool from its write counterpart.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly tells when to use the tool: to collect a file produced by sandboxed code or to inspect what `write_sandbox_file` left behind. It does not explicitly state when not to use it or name alternatives like `list_sandbox_dir` or `run_command`, but the context is clear enough to route usage.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_sandbox_infoA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden, and it delivers: it explains the two sandboxes are warmed per network policy, warmed independently, and that `warm: false` means the first call pays a ~26s download-and-compile cost. This goes beyond a generic status report and explains a key behavioral implication.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with no filler. The purpose is front-loaded, followed by concrete usage triggers and then the important warm/cold cost detail. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a zero-parameter status tool, the description is complete: it states what is reported, when to call it, and what a `false` result implies. An output schema exists, so return-value details do not need to be repeated in the description.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool takes zero parameters and the schema is empty, so there is nothing parameter-specific to explain. The baseline of 4 applies because the description instead adds useful context about what the report means.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description begins with a specific verb and resource: 'Report whether the sandboxes are warm, and what they are running.' This clearly identifies it as a status/diagnostic tool and distinguishes it from the mutating and file-operation sibling tools like reset_sandbox and write_sandbox_file.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit triggers: use it after a `warming` response, after unexpected slowness, or before a long task. It does not name alternatives or exclusion cases, but no direct alternative exists among the siblings, so the guidance is clear and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_commandA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral disclosure burden and does so exceptionally. It reveals that the sandbox is long-lived, that prior files and packages are visible, that side effects persist until reset_sandbox, and critically that args is a list and shell splitting/quoting/piping do not occur.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficient and front-loaded. Each sentence contributes: purpose, usage context, parameter clarification, return-shape reference, and long-lived side-effect behavior. There is no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a command-execution tool with no annotations and zero schema descriptions, the description covers the essential context: what it runs, when to use it, how args behaves, return shape, and sandbox persistence. It is slightly incomplete on cwd, env, network, and timeout_s semantics, but those are comparatively self-explanatory and the output schema covers return details.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It does add important meaning for command and especially args — 'not a shell string: there is no shell splitting, quoting or piping' — but cwd, env, network, and timeout_s are left undocumented in both the schema and description. The coverage is partial: 2 of 6 parameters receive substantive explanation.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Run any command available inside the Wasmer sandbox, with arguments.' It then clarifies scope by positioning the tool for non-Python work and explicitly contrasting it with run_python, making it easy for an agent to distinguish from siblings like run_python and reset_sandbox.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly states when to use the tool: 'Use it for non-Python work in the sandbox — a shell utility, or a binary that install_sandbox_package added.' This gives a clear selection rule and names the related sibling install_sandbox_package, and the mention of run_python's result shape reinforces the alternative.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_pythonA
  4.6/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does it thoroughly. It discloses the long-lived sandbox, persistence of files and packages, the separate network sandbox state, the return schema, and the important fact that a non-zero exit_code is a normal result rather than an error.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is organized into focused paragraphs: purpose, usage context, network behavior, output semantics, and persistence. Every sentence adds operational value and the most important scoping statement is front-loaded. Despite its length, no content is redundant.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description is highly complete for a stateful sandbox tool: it explains execution, safety, network mode, return values, persistence, and reset path. The main gap is the absence of any guidance on stdin and timeout_s behavior, which are the two underspecified parameters in an otherwise excellent descrption.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 0%, so the description must compensate. It adds real meaning for code (passed to python -c) and network (routes to a separate sandbox with host network), but it does not explain stdin or timeout_s. Those parameters are still largely inferable from their names, defaults, and types, but the description could have been even clearer.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the verb (execute), resource (Python code in an isolated Wasmer sandbox) and outcome (get back its output). It also differentiates from siblings by naming Python-specific execution and describing safety isolation, so an agent can distinguish it from run_command or sandbox file operations.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says when to use the tool: for untrusted snippets, experiments, quick calculations, and anything with side effects you want contained. It gives conditional guidance for network=true and mentions reset_sandbox for a clean slate, but it does not explicitly contrast with siblings like run_command or list_sandbox_dir, so no exclusions are stated.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• write_sandbox_fileA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden and does so well. It discloses that files are overwritten without asking, parent directories are created, the network flag selects a specific sandbox, sandboxes do not share filesystems, and the call returns the path and byte count.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is efficiently composed, with no filler. It front-loads the core purpose, then covers key usage details, behavioral caveats, and return information in a compact sequence.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity and the presence of an output schema, the description is complete. It covers purpose, parameters, sandbox isolation semantics, overwrite behavior, and return value, leaving little ambiguity for an agent.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 0%, so the description must compensate. It explains path should be absolute, provides an example, and clarifies the network flag's role in sandbox selection. Content is implied as the text to write but is not explicitly labeled, which prevents a perfect score.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool writes a text file inside the sandbox, creating parent directories as needed. It names specific use cases (script, fixture, input data) and distinguishes itself from sibling read/list tools by focusing on writing.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear context for when to use the tool: to place files where sandboxed code can read them, with the same filesystem visible to run_python and run_command. It also explains the network flag selects which sandbox, but it does not explicitly name alternatives or state when not to use this tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• reset_sandboxA
  4.8/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses the destructive nature ('throw away accumulated state'), the selective network reset behavior, and the asynchronous warm-up that may cause the next call to answer 'warming'. This is excellent behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences deliver the core action, usage triggers, parameter guidance, and an async caveat. It is front-loaded with the essential behavior and contains no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given no annotations and no output schema, the description provides everything needed to call the tool correctly: what it does, when to use it, how the parameter behaves, and expected asynchronous behavior. The mention of 'warming' covers an important operational expectation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema only defines network as string/null with a default. The description adds crucial meaning: pass 'off' or 'host' to reset that one, or omit to reset both. This fully explains the parameter's semantics beyond the schema.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action: throw away accumulated state and start fresh. It names the resource (sandbox) and distinguishes this tool from siblings like run_command or file operations by focusing on state reset.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives concrete conditions for use: earlier files, installed packages, or environment changes getting in the way, or when the sandbox behaves oddly. It does not explicitly mention when not to use it or name alternatives, but the context is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.