Server Quality Checklist

Tool Scores

• get_job_resultA
  4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden. It discloses the return shape (exit code, stdout, stderr, duration) and a key blocking behavior (waits for running jobs bounded by the job's timeout). It does not cover timeout failures or cancellation scenarios, but the disclosed traits are valuable.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Two concise, front-loaded sentences: the first states the tool's purpose, the second supplies essential return and blocking behavior. No filler or redundant content.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter tool with an output schema, the description covers the essential context: what it returns, when it blocks, and how it relates to execute_experiment. It could explicitly mention that job_id comes from execute_experiment, but this is a minor gap.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema fully describes job_id with 100% coverage. The description adds no extra parameter-level detail beyond the schema, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb ('Fetch') and resource ('completed result of a background command'). It implicitly distinguishes from get_job_status by focusing on result delivery rather than status, though it does not explicitly name the sibling alternative.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Provides clear context: the tool retrieves results of a background command and waits if the job is still running. However, it does not explicitly state when to prefer get_job_status or cancel_job, so no direct exclusion or alternative guidance is given.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• list_experimentsA
  4/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the burden of behavioral disclosure. It does reveal that results are ordered most recent first and that experiments are persistent resources that should be destroyed, which is useful context. However, it does not explicitly state that this is a read-only operation or describe any other behavioral nuances such as default status handling.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is extremely concise, with the core purpose in the first sentence and practical usage guidance in the second. Every sentence adds distinct value, and the structure front-loads the most important information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  This is a simple listing tool with optional parameters and an output schema, so the description does not need to explain return values. It gives the ordering and the intended use cases. A slight gap is the absence of explicit guidance about when to prefer get_experiment for single-item lookup, but overall the tool is sufficiently contextualized.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already provides 100% documentation coverage for both parameters, including defaults, constraints, and examples. The description adds no additional parameter-level meaning beyond what the schema contains, so the baseline score of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a clear verb and resource: 'List experiments', and adds the ordering 'most recent first'. This distinguishes it from sibling tools like get_experiment, which retrieves a single experiment. The use-case reference to sandboxes also makes the tool's purpose concrete.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says when to use the tool: to find a previously created sandbox or to check for undestroyed experiments before creating another. It does not explicitly name alternatives or exclusion conditions, so it falls short of the strongest usage guidance, but the context is clear.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_experimentA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral disclosure burden. It clearly indicates this is a read-style 'Fetch' operation and enumerates the state and summary contents returned: status, base image, isolation settings, resource limits, command counts, test summary, change statistics, and artifact count. This gives the agent a strong sense of what to expect without requiring the output schema.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact, front-loaded with the core action, then gives concrete usage guidance, and finally a useful return summary. Every sentence earns its place, and the bullet-like return list improves scannability without padding.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a single-parameter read tool with an output schema, the description is complete: it explains when to use it, what it does, and what kind of information it returns. There are no significant gaps that would prevent an agent from selecting and invoking this tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The single parameter experiment_id is already 100% described by the input schema ('The experiment to describe.'). The tool description adds no additional semantic detail about the parameter, so the baseline score of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Fetch an experiment's full state and a summary of what happened in it.' Its return list makes the purpose concrete, but it does not explicitly differentiate from siblings like get_job_status or check_sandbox_runtime, so it stops short of full sibling-level clarity.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides explicit use cases: 'USE THIS to re-orient -- after a long gap, or to check whether a sandbox is still alive before sending more commands.' This is clear context for when to call it, but it does not name alternatives or state when not to use it, so it misses the strongest form of usage guidance.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• inspect_changesA
  4.1/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden of explaining behavior. It explicitly states what the tool returns (created/modified/deleted file lists, insertion/deletion counts, optional unified diff) and mentions an important behavioral detail ('Build output and dependency directories are excluded'). It does not explicitly state that it is read-only, but the language implies a non-destructive inspection, which is transparent enough.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is exceptionally concise—three sentences that cover purpose, usage guidance, and output/exclusion details. Each sentence serves a distinct informative purpose with no redundancy. The structure flows logically from what it does, to when to use it, to what it returns.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's simplicity (3 parameters, 1 required, output described), the description is fully complete. It states the action, usage context, return format, and a key exclusion (build/dependency directories). There is no missing information that would prevent an agent from using the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  All three parameters have schema descriptions with 100% coverage, so the baseline is 3. The descriptions are clear: include_diff, experiment_id, and max_files_with_diff are self-explanatory. The tool description adds no additional parameter-level context beyond what the schema already provides, so it neither improves nor degrades the baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the primary action: showing what the experiment changed relative to the copied project. It also clarifies the scoped perspective ('against the project as it was copied in'), making the purpose unambiguous. However, it does not explicitly contrast with sibling tools, which would elevate it to a 5.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description provides direct usage guidance: 'USE THIS before destroying a sandbox, and before telling the developer what you found.' This gives concrete scenarios when the tool is appropriate. It lacks an explicit 'when not to use' or alternative tool recommendation, but the provided triggers are specific and actionable.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• compare_experimentsA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden, and it does well: it discloses the return fields (test results, failing test names, exit codes, change size, duration, artifact counts), conditional recommendation behavior, and that destroyed experiments remain comparable. It does not explicitly state whether the operation is read-only or whether comparing triggers any side effects, but the described behavior is otherwise transparent.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured with clear sections: a terse one-line purpose, a concrete 'USE THIS' scenario, and a focused 'RETURNS' list. Every sentence earns its place, and the most decision-relevant information is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The tool has moderate complexity, an output schema, complete parameter schema coverage, and no annotations. The description covers purpose, usage context, return behavior, and the destroyed-experiment edge case. It could be more complete by explicitly stating that this is a read-only analytic comparison that does not run or modify experiments, but it is otherwise sufficient for an agent to select and call it correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100%, so the input schema already documents experiment_ids and labels with examples. The description adds context about comparing destroyed experiments, which relates to how experiment_ids may be interpreted, but it does not add meaningful parameter-level semantics beyond what the schema provides. Baseline 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool compares two or more experiments side by side, with a specific verb ('compare') and resource ('experiments'). It also names the core differentiating output (per-experiment results plus a recommendation), which separates it from siblings like get_experiment, list_experiments, and run_tests.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives an explicit scenario: use this when several approaches were tried and a recommendation is needed. It advises running each approach in its own experiment first. However, it does not explicitly state when NOT to use it or name alternative tools such as run_tests or get_experiment for simpler lookup needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• get_job_statusA
  4.2/5.0
  Behavior3/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden. It discloses read-only polling semantics, the elapsed-milliseconds return, and a terminal condition (`finished`). However, it does not specify the status vocabulary or failure-mode behavior — e.g., whether a failed or cancelled job ever reaches `finished`, which is a clear gap for an agent polling a background job.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Three sentences, no filler: purpose first, then return values, then the polling next-step. Every sentence earns its place and the key behavioral instruction is front-loaded.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter, read-only polling tool with an output schema present, the description covers the core workflow completely: what it checks, what it returns, and what to do when done. The only meaningful omission is guidance for terminal states other than `finished` (failure/cancellation), but the output schema likely documents status values.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and job_id is already documented as 'Job id from execute_experiment.' The description adds contextual value by tying job_id to commands started with background=true, but it doesn't add parameter-specific syntax or format details beyond the schema, so the baseline of 3 is appropriate.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb and resource: 'Check on a command started with background=true.' It also says what it returns (status, elapsed milliseconds), and the polling-then-result flow clearly distinguishes it from get_job_result, which retrieves the job's output rather than its status. The scope is precise and disambiguating.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Gives explicit usage flow: poll this for background commands, wait until `finished` is true, then switch to get_job_result. It names the sibling tool and the exact condition for switching, which is exactly the routing guidance an agent needs.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• read_sandbox_fileA
  4.2/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral disclosure burden. It states the tool is read-only, returns text, uses workspace-relative paths, and cannot escape the sandbox. It does not describe error behavior for missing files or binary files, but for a simple read tool the key safety and operational constraints are disclosed.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is tight and front-loaded. The first sentence states the core action, the second gives a concrete use case, and the third states a critical safety limitation. No sentence is wasted.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's low complexity, high schema coverage, and presence of an output schema, the description is largely complete. It explains what, when, and the security boundary. Minor gaps like error behavior for nonexistent paths are left to the output schema and runtime errors.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3. The description reinforces that paths are workspace-relative and cannot escape, which matches the schema's existing path description. It adds little beyond the schema for experiment_id, but no parameter meaning is left unclear.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Read a file from inside the sandbox as text.' It clearly distinguishes the tool from write_sandbox_file and other sibling operations by framing it as an investigation tool for reading source, config, or logs.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit usage guidance: 'USE THIS to investigate a failure -- read the source, the config, a log -- without guessing from stack traces.' It also explains an important exclusion, noting it cannot read the developer's machine. It does not explicitly name a sibling alternative, but the read vs. write distinction is clear enough.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• cancel_jobA
  4.3/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the behavioral burden. It discloses that the tool stops a running command, kills its process, and is idempotent on finished jobs by returning the final state untouched. This gives the agent a clear mental model of side effects and edge-case safety.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is compact and front-loaded with the action, followed by a direct usage criterion and an idempotency note. Every sentence earns its place with no filler or repetition.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a one-parameter tool with an output schema, the description is sufficiently complete. It covers the core action, the circumstances that warrant cancellation, and the behavior on already-finished jobs, leaving no critical gap for correct invocation.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100% and the sole parameter 'job_id' is already described as 'Job id to cancel.' The description adds no additional parameter-level meaning, but none is needed given the high schema coverage and single straightforward parameter.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description uses a specific verb and resource: 'Stop a running command and kill its process inside the sandbox.' This clearly identifies the tool's purpose and distinguishes it from sibling tools that check status or retrieve results.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says when to use the tool: 'USE THIS when a command is clearly stuck or no longer needed.' It also gives important guidance about safety on already-finished jobs, though it does not explicitly name alternative tools or when-not conditions.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• collect_artifactsA
  4.4/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden, and it delivers useful behavioral context: files land in the server's state directory, the tool cannot write on the developer's machine, and it must be used before sandbox destruction. It stops short of explaining edge-case behavior such as no matching patterns, but the disclosed storage location and limitations are meaningful and non-obvious.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is three sentences with no filler. The purpose is front-loaded, use cases are in the second sentence, and the critical path/pattern details are in the third. Every sentence contributes new information.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a two-parameter tool with an output schema present, the description covers the core purpose, when to use it, path semantics, output location, and an important limitation. It does not explicitly mention sibling alternatives or what happens if a pattern matches nothing, but an agent has enough information to select and call the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already documents both parameters at 100% coverage. The description adds value by detailing pattern semantics beyond the schema: workspace-relative shell globs, the '**/name' recursive form, and concrete examples. The experiment_id parameter is not further enriched, but the schema already explains it sufficiently.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb+resource: 'Copy selected files out of a sandbox before it is destroyed.' This clearly identifies what the tool does and emphasizes the critical timing constraint. It also distinguishes itself from siblings like read_sandbox_file by describing a copy-out operation rather than inline reading.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit use cases: 'USE THIS for build output, test reports, coverage, benchmark results or logs you want to keep or quote.' This is strong when-to-use guidance. It does not explicitly name alternatives or state when not to use it, but the timing constraint 'before it is destroyed' and the restriction about not writing to the developer's machine provide clear operational context.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• execute_experimentA
  4.4/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the full burden and does so thoroughly: it clarifies commands run in a container/workspace, never on the host; non-zero exits are normal results; background=true changes behavior; and timeout bounds prevent hangs. This is rich, honest behavioral context.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  Four compact paragraphs, each with a distinct job: core action, usage guidance, return behavior, and background execution. The most important information is front-loaded and every sentence adds useful guidance without redundancy.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers execution environment, return values, non-zero exit semantics, timeout behavior, and the background workflow. Combined with the 100% schema coverage and the presence of an output schema, an agent has enough context to select and invoke the tool correctly.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already covers 100% of parameters, so the baseline is 3. The description adds value beyond the schema by explaining how background=true should be used (poll get_job_status, fetch get_job_result) and by clarifying that timeout bounds execution. This elevates it above the baseline.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose4/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  States a specific verb and resource ('Run a shell command inside a sandbox') and gives concrete examples like installs, builds, scripts, and migrations. It is clear what the tool does, but it does not explicitly distinguish itself from the sibling run_tests, which could overlap in the 'run things in the terminal' space.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Gives explicit when-to-use guidance: 'USE THIS for anything you would otherwise run in the developer's terminal.' It also describes the background workflow with get_job_status and get_job_result. However, it does not state exclusions or directly compare against alternatives like run_tests.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• run_testsA
  4.6/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations, the description carries the burden of explaining behavior, and it does so well: auto-detects the runner, parses results, returns exit code/stdout/stderr/duration, and warns to trust exit code over zeros when test_summary.detected is false. It does not explicitly address side effects, but the sandbox context implies isolation.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is well-structured into purpose, usage guidance, and return behavior, with no filler. Every sentence adds useful information, and the line breaks improve readability.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness4/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's moderate complexity—auto-detection, command override, parsed test summaries—the description is complete enough for an agent to call it correctly. It does not detail output schema fields, but it sufficiently describes what is returned, so no critical context appears missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The schema already covers all three parameters with clear descriptions, and the tool description adds the important note that command overrides detection. experiment_id, timeout, and command are all semantically meaningful with no ambiguity.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool runs the project's test suite inside a sandbox and parses results. It also explicitly contrasts with execute_experiment by framing this as the way to determine whether the project still works, which distinguishes it from siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  It gives explicit guidance to use this instead of execute_experiment when the goal is checking whether the project still works. It also explains that command can override auto-detection, giving concrete usage direction.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• create_experimentA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden of behavioral disclosure, and it delivers thoroughly: snapshot-copy semantics, no propagation back to the developer's tree, secrets withheld, network disabled by default, environment variables refused unless allowlisted, credential-shaped names rejected, and resource caps. It also tells the agent to read the warnings field for clamped settings.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is information-dense but every sentence earns its place. It is front-loaded with the core definition, uses clear section markers for usage, returns, and safety, and the length is justified for a safety-critical tool with 10 parameters and no annotations.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  Given the tool's complexity, 10 parameters, zero annotations, and presence of an output schema, the description is complete. It covers creation semantics, safety properties, usage timing, return behavior, warning handling, and resource restrictions. Nothing an agent needs to decide whether and how to invoke create_experiment is missing.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters4/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema coverage is 100%, so the baseline is 3, but the description adds meaningful cross-parameter behavioral context: it explains how project_path is snapshot-copied, how environment_allowlist is enforced, how network_mode defaults to disabled, and how cpu/memory limits are clamped. This goes beyond what the individual parameter descriptions state.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description opens with a specific verb and resource: 'Create a disposable, isolated environment and copy a project into it.' It clearly identifies the tool's core purpose and scope, distinguishing it from siblings like execute_experiment, get_experiment, and destroy_experiment.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines4/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit guidance on when to use this tool: before risky operations like installing dependencies, running builds, migrations, or unfamiliar code. It warns to 'reach for it before the risky step, not after,' but does not explicitly name alternative tools or exclusion cases, so it falls just short of a 5.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• destroy_experimentA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are provided, so the description carries the full behavioral burden and meets it: it discloses the destructive nature, idempotency ('calling it on an already-destroyed experiment is safe and returns the stored report rather than an error'), and the return contract ('what changed inside the sandbox'). This is exactly the disclosure an agent needs for a destructive call.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is front-loaded with the core action and organized into clearly labeled sections (usage directive, idempotency, returns). Every sentence earns its place by adding a distinct piece of information — there is no filler, repetition, or vague phrasing.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  For a destructive single-parameter tool with no annotations, the description is complete: it states the action, when to call it, the cost of not calling it, the idempotent re-call behavior, and the contents of the return. An output schema exists, so the return structure itself is already covered structurally.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  Schema description coverage is 100% — the sole parameter experiment_id is already documented as 'The experiment to destroy.' The description adds no parameter-specific detail beyond the schema, so the baseline of 3 applies.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource — 'Destroy a sandbox and everything in it' — and adds the outcome ('returning a final report'). This clearly differentiates it from siblings like get_experiment (read-only), execute_experiment (run), and cancel_job (cancel a job, not destroy the sandbox).

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description gives explicit when-to-use guidance: 'USE THIS as soon as an experiment has told you what you needed. Always call it.' It also supplies the rationale (a sandbox left running consumes CPU and memory) and implicitly tells the agent not to call it prematurely — collect what you need first, then destroy.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• write_sandbox_fileA
  4.7/5.0
  Behavior5/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  With no annotations provided, the description carries the full burden, and it delivers: it discloses that writes are sandbox-only, that no tool writes to the developer's project, and that parent directories are created as needed. This is unusually clear behavioral context for a mutating tool.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is tight and well-structured: the action is first, followed by when to use it, then a safety note. Each sentence earns its place with no redundancy or filler.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description covers purpose, usage context, safety boundaries, and a key behavioral guarantee. Since the schema covers all parameters and an output schema exists, nothing essential is missing for an agent to correctly select and invoke this tool.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters3/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The input schema already documents all three parameters with 100% coverage. The description adds very little parameter-specific meaning beyond referencing the sandbox context, so the baseline of 3 is appropriate because the schema does the heavy lifting.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description states a specific verb and resource: 'Write a file inside the sandbox.' It also adds a distinct behavioral detail, creating parent directories, and frames the tool's purpose as applying a candidate fix, clearly separating it from read_sandbox_file and other siblings.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  The description explicitly says 'USE THIS to apply a candidate fix' and instructs to prefer it over shell heredocs to avoid quoting issues. It also gives a clear when-not-to-use rule: writes only affect the sandbox, so permanent changes should be shown via inspect_changes and applied by the developer.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.

• check_sandbox_runtimeA
  4.8/5.0
  Behavior4/5

  Does the description disclose side effects, auth requirements, rate limits, or destructive behavior?

  No annotations are present, so the description carries the burden. It conveys a read-only operation ('Confirm', 'report', 'Returns') and does not imply any side effects. However, it does not explicitly state 'does not modify state', leaving a small gap in transparency.

  Agents need to know what a tool does to the world before calling it. Descriptions should go beyond structured annotations to explain consequences.

  Conciseness5/5

  Is the description appropriately sized, front-loaded, and free of redundancy?

  The description is two sentences, front-loaded with the core purpose followed by a precise usage scenario. Every sentence is necessary and there is no fluff.

  Shorter descriptions cost fewer tokens and are easier for agents to parse. Every sentence should earn its place.

  Completeness5/5

  Given the tool's complexity, does the description cover enough for an agent to succeed on first attempt?

  The description fully covers what the tool does, when to use it, and what it returns. Even without an output schema, the explicit mention of 'Docker version' and 'isolation defaults' gives the agent enough context to interpret the result.

  Complex tools with many parameters or behaviors need more documentation. Simple tools need less. This dimension scales expectations accordingly.

  Parameters5/5

  Does the description clarify parameter syntax, constraints, interactions, or defaults beyond what the schema provides?

  The tool has zero parameters, so the description cannot add parameter-level meaning beyond the schema. The baseline for 0 params is 4, but the description adequately covers the absence of inputs without needing extra clarification, thus earning a 5.

  Input schemas describe structure but not intent. Descriptions should explain non-obvious parameter relationships and valid value ranges.

  Purpose5/5

  Does the description clearly state what the tool does and how it differs from similar tools?

  The description clearly states the tool's purpose: 'Confirm the sandbox runtime is available and report the active defaults.' The verb and resource are specific, and it is easily distinguished from sibling tools like create_experiment or get_job_status.

  Agents choose between tools based on descriptions. A clear purpose with a specific verb and resource helps agents select the right tool.

  Usage Guidelines5/5

  Does the description explain when to use this tool, when not to, or what alternatives exist?

  Explicit guidance is provided: 'USE THIS first if create_experiment fails' and explains the diagnostic value (distinguishing a stopped Docker daemon from a rejected request). This tells the agent exactly when to invoke this tool over alternatives.

  Agents often have multiple tools that could apply. Explicit usage guidance like "use X instead of Y when Z" prevents misuse.