Server Quality Checklist

Tool Scores

• get_job_resultA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the burden and does meaningful work: it discloses that the call waits for a running job and is bounded by the job's timeout, and it specifies the returned fields (exit code, stdout, stderr, duration). It does not cover error/timeout outcomes, but for a simple retrieval-and-wait tool this is solid behavioral disclosure.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two sentences with no filler: the first states the core purpose, and the second provides the key behavioral detail and return shape. All information is front-loaded and every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter tool with an output schema, the description is largely complete: it names the resource, explains blocking behavior, and references the return shape. It could note where job_id comes from or how timeout/failure behaves, but these are minor gaps given the simplicity and output schema coverage.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The only parameter, job_id, is fully documented in the schema with 'Job id to fetch.' and schema description coverage is 100%. The description does not add further parameter-level guidance, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description opens with a specific verb and resource: 'Fetch the completed result of a background command.' It also clarifies what it is not by noting it returns the same shape as execute_experiment, which differentiates it from status-only siblings like get_job_status. The purpose is immediately clear.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines3/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description implies usage: call this to get a background command's result, and it will wait if the job is still running. However, it does not explicitly mention alternatives such as get_job_status for non-blocking status checks or say when not to use this tool. The context is clear but the boundary against siblings is left to inference.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_experimentsA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral burden. 'List' clearly indicates a non-mutating read, and 'most recent first' plus the sandbox-lifecycle hint add useful context. It does not explicitly mention side effects or permissions, but for a simple read-only list the risk of ambiguity is low.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two compact sentences with no filler. The core behavior is front-loaded in the first sentence, and the second sentence adds practical usage context without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given that both parameters are fully documented in the schema and an output schema exists, the description provides the missing decision context: when to use this tool and what ordering to expect. Nothing critical is missing for a list-only operation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already documents both parameters fully, including defaults, ranges, and meaning, so the description does not need to repeat them. The description adds no parameter-specific detail, but the baseline 3 applies because schema coverage is 100%.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the action ('List'), the resource ('experiments'), and the ordering ('most recent first'). The plural resource and the sandbox-finding use case distinguish it from singular or CRUD siblings like get_experiment and create_experiment, though it does not explicitly name a competing tool.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly tells the agent when to use this tool: to find a sandbox created earlier or to check for undestroyed sandboxes before creating another. It lacks an explicit 'when not to use' or alternative-name mention, which keeps it from a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cancel_jobA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses that cancellation kills the process and that calling it on a finished job is safe and returns the final state untouched. This covers the key behavioral expectations, though it does not mention side effects like partial artifacts or queued jobs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured. It leads with the core action, then gives a direct usage condition, and closes with an important safety note. No wasted words.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has one simple parameter and an output schema, so the description does not need to explain return values. It sufficiently covers action, usage, and safety. Some edge cases, such as cancelling a queued versus running job, are not addressed, keeping it just short of a 5.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already documents job_id as 'Job id to cancel' with 100% coverage. The description does not add additional meaning about the parameter, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses specific verbs ('stop', 'kill') and names the resource ('a running command inside the sandbox'). It clearly differentiates this from sibling tools like get_job_status or get_job_result by focusing on cancellation rather than inspection.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says when to use it: when a command is 'clearly stuck or no longer needed'. It also gives a helpful safety condition about already-finished jobs. It does not explicitly name alternatives or exclusions, so it misses the top score.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_experimentA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  There are no annotations, so the description carries the behavioral burden. 'Fetch' implies a read-only operation, and the description clearly lists the state and summary data returned, including liveness. It does not explicitly state side-effect-freeness or permissions, but nothing suggests mutation or hidden costs.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and well-structured: one sentence for what it does, one for when to use it, and one for what it returns. Every sentence earns its place with no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given a single parameter, an output schema, and no annotations, the description is largely complete: it covers purpose, return content, and a representative use case. A short note about which sibling tools to prefer for narrower needs would make it fully complete.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% and the single parameter experiment_id is already described as 'The experiment to describe.' The description adds no new parameter-level detail, which matches the baseline for fully covered schemas.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses the specific verb 'Fetch' with a clear resource ('an experiment's full state and a summary of what happened in it') and enumerates the returned items. This makes it easy to distinguish from sibling tools like list_experiments or inspect_changes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly gives use cases: re-orienting after a long gap and checking whether a sandbox is still alive before more commands. It does not name alternatives or exclusions, such as pointing to check_sandbox_runtime for a lighter liveness check, so it stops short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_experimentsA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral disclosure burden. It is transparent about the return payload (test results, failing tests, exit codes, change size, duration, artifact counts, conditional recommendation) and explicitly notes that destroyed experiments remain comparable because findings were recorded pre-teardown.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core purpose, then uses clear 'USE THIS' and 'RETURNS' signals to organize usage and behavior. Every sentence contributes distinct information; the examples make the usage guidance concrete without padding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the simple two-parameter schema and presence of an output schema, the description covers the decision context, prerequisites implied by sibling tools, and the key edge case (destroyed experiments). Nothing critical is missing for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents both parameters (experiment_ids, labels). The description adds no further parameter-level detail, which is acceptable at the baseline for fully documented schemas.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Compare two or more experiments side by side.' This clearly distinguishes it from sibling tools like create_experiment, get_experiment, and list_experiments.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The 'USE THIS when...' section gives explicit scenarios (multiple candidate fixes, runtime versions, dependency upgrades) where the tool should be chosen. It does not enumerate alternatives or explicit when-not-to-use cases, but the context is unambiguous.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_job_statusA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full behavioral burden. It discloses that the tool returns status and elapsed milliseconds, that it is designed for polling, and that completion is signaled by the finished flag. This adequately conveys the non-terminal, read-only nature of the operation without contradicting anything.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the core purpose, followed by return values and the next step. Every sentence earns its place, and there is no redundant or filler content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple one-parameter status-polling tool with an output schema, the description fully covers purpose, usage, return values, and the subsequent call to get_job_result. Nothing essential is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already provides 100% coverage with a clear description of job_id as coming from execute_experiment. The tool description adds minimal extra parameter context beyond implying the job is a background command, so it meets the baseline but does not exceed it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action — checking on a command started with background=true — and clearly identifies the target resource (a job). It also distinguishes itself from get_job_result by describing the polling-to-result handoff, so an agent can tell the two sibling tools apart.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives clear when-to-use guidance: use it for background commands, poll it, then call get_job_result once finished is true. It does not explicitly state exclusions or alternatives beyond the get_job_result handoff, but the context is strong enough for correct selection.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspect_changesA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses return contents (created/modified/deleted files, counts, optional diff) and filtering behavior (build output and dependency directories excluded). It does not explicitly state that the tool is non-destructive, which would make this a 5.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact, front-loaded with the core purpose, and every sentence earns its place: what it does, when to use it, and what it returns. The node_modules example efficiently justifies the exclusion behavior.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description tells the agent why this matters, when to invoke it, what baseline is used, what outputs to expect, and how noise is filtered. The presence of an output schema covers the return structure, so nothing important is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters. The description's mention of 'optionally the unified diff' aligns with include_diff but adds little beyond the schema. Baseline 3 is appropriate because the schema handles parameter meaning.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Show what the experiment changed, against the project as it was copied in.' It clearly distinguishes this from siblings like compare_experiments or get_experiment by defining the baseline and the evidence role.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit usage timing: 'USE THIS before destroying a sandbox, and before telling the developer what you found.' It does not explicitly name alternatives or when-not-to-use, so it stops short of a 5, but the usage context is strong.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• collect_artifactsA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of behavioral disclosure. It explains that files land in the server's state directory, that the tool cannot write to the developer's machine, and that it is meant to run before sandbox destruction. It stops short of detailing edge cases like empty pattern matches or overwrite behavior, but the core side effects are transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, each earning its place: the core action, the primary use cases, and the pattern/destination details. The most important information is front-loaded, and there is no redundant restating of the tool name or schema.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a tool with only two required parameters and an output schema, this description is complete. It tells the agent what the tool does, when to use it, how patterns work, where files go, and key constraints. No annotations are provided, but the description compensates with sufficient behavioral context.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and parameter descriptions are already present. The description adds meaningful glob semantics beyond the schema: workspace-relative patterns, examples like 'dist/*.js', and the '**/name' recursive-search convention. This helps the agent construct valid inputs.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Copy selected files out of a sandbox before it is destroyed.' This clearly distinguishes it from siblings like read_sandbox_file, which reads a file without copying it out, and get_job_result, which returns job outputs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'USE THIS for build output, test reports, coverage, benchmark results or logs you want to keep or quote,' giving the agent clear triggering conditions. It also states a limitation ('cannot write anywhere on the developer's machine') that implies when not to use it, though it does not explicitly name an alternative tool.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_experimentA
  4.5/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations present, the description carries the full burden of behavioral disclosure. It does so thoroughly: commands run in the container, never the host; returns exit code, stdout, stderr, duration; a non-zero exit is a normal result; background mode returns a job id; and the call is bounded by timeout so it cannot hang the session. This gives the agent a complete mental model of the tool's behavior.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is tightly organized: purpose, usage guidance, return behavior, and background-mode instructions. Every sentence adds value, and the most important information is front-loaded. It remains concise despite covering a lot of behavioral ground.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has 5 parameters, an output schema, and a diverse sibling set. The description covers the core function, return values, error semantics, timeout behavior, background job handling, and boundary (sandbox vs host). It gives the agent everything needed to call this tool correctly and react to results, with no significant gaps.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the baseline is 3. The description adds some context around background=true (poll get_job_status, fetch get_job_result) and timeout ('bounded by the timeout'), but the schema already documents each parameter adequately. The description does not materially increase parameter understanding beyond what the schema provides.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Run a shell command inside a sandbox.' This clearly distinguishes it from sibling tools like read_sandbox_file, write_sandbox_file, and run_tests by framing it as a general-purpose command executor.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It explicitly says 'USE THIS for anything you would otherwise run in the developer's terminal' and gives concrete examples (installs, builds, scripts, migrations). It also provides guidance for long-running commands via background=true and mentions polling get_job_status and get_job_result. It does not explicitly contrast with run_tests or inspect_changes, so it misses the 'when not to use' part, but the context is otherwise very clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_sandbox_fileA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden. It discloses key behavioral traits: paths are workspace-relative, cannot escape the sandbox, and the tool returns text content. This goes beyond the schema and helps the agent understand safety boundaries, though it does not cover error behavior or binary file handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two tight sentences with no filler. It front-loads the core action, then immediately provides the primary use case and critical constraints. Every sentence earns its place.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple two-parameter read tool with an output schema available, the description is complete. It explains what the tool does, when to use it, and the key security boundary. No critical invocation details are missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds meaning to the 'path' parameter by specifying that paths are workspace-relative and cannot escape the sandbox, which is not fully apparent from the schema alone. The 'experiment_id' parameter is not elaborated beyond the schema, but the added path context is valuable.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description starts with a specific verb and resource: 'Read a file from inside the sandbox as text.' It also distinguishes from siblings by stating that paths cannot escape the sandbox and that the tool cannot read the developer's machine, making its scope unambiguous.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance: 'USE THIS to investigate a failure' and suggests reading source, config, or logs rather than guessing from stack traces. It also notes a limitation (cannot read the developer's machine), but it does not explicitly name alternative tools or when-not-to-use conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• write_sandbox_fileA
  4.5/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the transparency burden and does well by stating writes land 'in the sandbox copy only' and that 'There is no tool here that writes to the developer's project.' It also discloses automatic parent directory creation. It does not explicitly state overwrite behavior, but the schema's 'Full new contents' parameter description implies replacement.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three short, purposeful sections: what it does, when to use it, and safety scope. Every sentence earns its place, and the core function is front-loaded before usage and safety notes.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a simple 3-parameter tool with fully described schema fields and an output schema, the description covers the action, side effects, use context, and safety boundary. Nothing essential for an agent to invoke it correctly is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema fully documents all three parameters. The description adds minimal parameter-specific meaning beyond the schema, only noting parent directory creation for path and positioning content as a candidate fix. This meets the baseline but doesn't elevate it.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  Description opens with 'Write a file inside the sandbox' - a specific verb and resource that clearly distinguishes it from read_sandbox_file and other siblings. It adds meaningful scope with 'creating parent directories as needed' and ties the tool to applying candidate fixes.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicitly says 'USE THIS to apply a candidate fix' and instructs preferring it over shell heredocs because 'no quoting to get wrong.' It also clarifies when not to use it for permanent changes, directing the user to show the diff from inspect_changes instead, since no tool writes to the developer's project.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_sandbox_runtimeA
  4.7/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the behavioral burden. It discloses that the tool returns the Docker version and isolation defaults, and frames the action as a non-mutating check. It could further clarify failure behavior, but for a zero-parameter diagnostic tool this is sufficient.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: the first sentence states the core purpose, and the second adds a valuable usage cue and return details. Every sentence earns its place with no repetition or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool is simple (0 params, output schema present), and the description fully covers when to use it, what it does, and what it returns. Nothing essential is missing for an agent to select and call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters and an empty input schema, so parameter documentation is not needed. The description correctly focuses on behavior and output rather than parameter details, matching the baseline for parameter-less tools.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's function: confirm sandbox runtime availability and report active defaults. It names the specific resource (sandbox runtime) and the exact outputs (Docker version, isolation defaults), distinguishing it from experiment-related siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit situational guidance: use this first if create_experiment fails to distinguish a stopped Docker daemon from a rejected request. This directly tells an agent when to invoke it over alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_experimentA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description fully carries the behavioral disclosure burden. It explains that the project is snapshot-copied, changes never propagate back, secrets and build output are withheld, network is disabled by default, environment variables are restricted, and CPU/memory/PIDs/wall-clock are capped. It also warns about the returned warnings field.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is detailed but well-structured and front-loaded: purpose, usage, isolation behavior, return value, and safety. Every sentence earns its place and the use of short labeled sections ('RETURNS', 'SAFETY') makes the content easy to scan.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a complex 10-parameter tool with no annotations, this description is exceptionally complete. It covers purpose, usage timing, data-flow guarantees, security boundaries, returned values, and limitations. The output schema exists, so return details are already structured, and the description still adds the key warning-field guidance.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the schema already documents every parameter. The description adds valuable behavioral semantics beyond the schema: credential-shaped environment variable names are refused, resource limits are clamped/capped, and network is off unless requested. This extra context enhances correct parameter usage.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Create a disposable, isolated environment and copy a project into it.' It clearly distinguishes this creation tool from its siblings by focusing on setup and isolation, and the later 'USE THIS when' section reinforces what this tool is for.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit, concrete usage guidance: use this before risky actions like installing dependencies, running builds, migrations, upgrades, or unfamiliar code. It strongly communicates the when, but does not explicitly name alternative tools or state when not to use it, so it falls just short of a perfect 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_testsA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full behavioral burden and does so thoroughly. It discloses the return contents (exit code, stdout/stderr, duration, parseable summary), and importantly warns that when test_summary.detected is false, the agent should trust the exit code rather than zero counts.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded: purpose first, then usage guidance, then return behavior, then the override hint. Every sentence adds distinct value, and there is no redundant filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool operates in a sandbox, has an output schema, and the description explains the key caveat about detection reliability. It covers purpose, alternatives, return semantics, auto-detection, and command override, making it complete for an agent to invoke correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the schema already documents all parameters well. The description adds minor reinforcement by saying 'Pass command to override detection,' but it does not substantially add meaning beyond the parameter descriptions already present.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific action ('Run the project's test suite inside a sandbox and parse the results') with a clear resource and outcome. It also distinguishes itself from execute_experiment by explicitly naming when this tool should be preferred, so an agent can tell them apart.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives an explicit usage rule: 'USE THIS instead of execute_experiment when you want to know whether the project still works.' It also explains the auto-detection behavior and how to override it with 'command', which gives the agent enough context to decide correctly.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• destroy_experimentA
  4.9/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full burden of behavioral disclosure. It clearly warns that the tool destroys the sandbox and everything in it, states that it is idempotent, and explains that calling it on an already-destroyed experiment returns the stored report instead of erroring.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with front-loaded purpose, then usage guidance, idempotency, and return-value explanation. Each sentence earns its place and there is no filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter tool with an output schema, the description is complete. It covers when to use it, what it does, the destructive scope, idempotency behavior, and what the return report contains.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description adds extra meaning beyond the schema by explaining that experiment_id may refer to an already-destroyed experiment and that the call remains safe, returning the stored report rather than failing.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Destroy a sandbox and everything in it.' It also clarifies the return value, a final report, which distinguishes it from sibling tools that create, execute, read, or inspect experiments. There is no ambiguity about what this tool does.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'USE THIS as soon as an experiment has told you what you needed' and 'Always call it,' giving a clear trigger and a strong directive. It also explains the downside of not calling it: a sandbox left running keeps consuming CPU and memory.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.